Discussion Forums

Hi,

I've created a command line tool using the ThinkGeo engine suite. It takes two shapefiles and merges them together.

Here is the code:

        ///

        /// Merges all the values in the merge layer object with those in the main lyaer object
        ///

        ///
        ///
        ///
        static void TransferData(Layer mainLyr, Layer mergeLyr, bool GCOnEveryRun)
        {
            if (!mainLyr.Editing)
            {
                throw new ApplicationException("Main layer has not been made editable");
            }

            DateTime start = DateTime.Now;
            ConsoleWriteLine("Transferring data from " + mergeLyr.ShapeName + " to " + mainLyr.ShapeName);

            int[] recIds = mergeLyr.SQLQuery(string.Format("select * from [{0}]", mergeLyr.ShapeName), "RECID");
            ConsoleWriteLine(recIds.Length + " record ids returned from " + mergeLyr.ShapeName);

            //get columns for data transfer from main layer
            List columns = new List();
            //remove any RECID field
            foreach (string colName in mainLyr.ColumnsArray)
            {
                if (colName != "RECID")
                {
                    columns.Add(colName);
                }
            }

            int insertComplete = 0;
            int insertCount = 1;
            BaseShape shp = null;
            //insert shapes from main shapefile into new shapefile
            foreach (int id in recIds)
            {

                shp = mergeLyr.GetShape(id);

                //outofmemoryexception occurs calling this method
                int shapeId = mainLyr.AddShape(shp);

                Object[] arr = (Object[])mergeLyr.DataQuery(new int[] { id }, columns.ToArray());
                string[] fieldValues = (string[])arr[0];

                mainLyr.UpdateTableData(shapeId, columns.ToArray(), fieldValues);

                //write to console
                insertComplete = (int)((100 / (Double)recIds.Length) * insertCount);
                Console.Write(String.Format("Inserted: {0} of {1} ({2}%)\r", insertCount, recIds.Length, insertComplete));
                insertCount++;

                if (GCOnEveryRun)
                {
                    //need to do this for very large files
                    GC.Collect();
                }
            }

            ConsoleWriteLine("Data transferred from " + mergeLyr.ShapeName + " (" + DateTime.Now.Subtract(start).ToString() + ")");
        }

The code works fine on almost all files up to a size of around 1.5gb.

There are two issues I am having:

1 - on windows XP machine with 3Gb RAM the tool errors on very large files (c. 1.6Gb) in size. The main shapefile is 1.6Gb in size and the merging file is only 20Mb in size. I get an outofmemoryexception when I call the mainLyr.AddShape() method.

2 - I have tried setting the /3GB switch in the boot.ini file on the machine to expand the memory space but this does not seem to work. So i've been trying to get the code to run on a 64-bit machine but I get the following error just calling the command line tool:

Unhandled Exception: System.BadImageFormatException: Could not load file or assembly 'EngineEdition, Version=2.0.0.0, Culture=neutral, PublicKeyToken=0828af5241fb4207' or one of its dependencies. An attempt was made to load a program with an incorrect format.

File name: 'EngineEdition, Version=2.0.0.0, Culture=neutral, PublicKeyToken=0828af5241fb4207' at Merge.Merge.Main(String[] args)

The code has been compiled using the x64 flagin the build configuration in VS.net but does not seem to make any difference.

So:

1 - Do you hav any ideas on how to make the files merge without throwing the outofmemoryexception

2 - Do you have a compiled 64-bit version of the engine edition?

Thanks in advance.

 

 

I would recommend braking the merging operation into pieces. 1.6 Gb is a huge for a shapefile and I think that if you process, for example, 100 000 records at a time and then commit, it is going to be less demanding on the memory.
For what operating system are you looking for a 64-bit verion of the Engine edition?

Thanks for respoonding Val.
I could try and commit after 100,000 records but from what I understand of the engine version a duplicate of the shapefile (_Editing.ashp) is created and records are pushed into this file. So committing after 100,000 records wouldn't make a difference would it? I am also calling the garbage collection after each insert so any leftover data in memory should be cleaned up. I will try and batch run it and let you know how I go.
As for the 64-bit version - windows Server 64-bit and also Vista 64-bit. IS there a 64-bit version?

Actually, thinking about it, the number of records being inserted into the 1.6Gb shapefile is only about 60,000. May try a smaller number and see what happens.
Thanks.

Let us know what you find.

Will do. Any news on the 64-bit version of MapSuite Engine? Is there one?

David,

Map Suite is not a native 64-bit control. Rather it can run in a 64-bit environment but only as a 32-bit control.

So how do I run the command line application, which uses the MapEngine componenet, and not get the following error on a 64-bit machine?:

Unhandled Exception: System.BadImageFormatException: Could not load file or assembly 'EngineEdition, Version=2.0.0.0, Culture=neutral, PublicKeyToken=0828af5241fb4207' or one of its dependencies. An attempt was made to load a program with an incorrect format.
File name: 'EngineEdition, Version=2.0.0.0, Culture=neutral, PublicKeyToken=0828
af5241fb4207'
at Merge.Merge.Main(String[] args)

Thanks.

David,

Did you set the compile option to x86 in your application, because the MapSuite map control is not a native 64-bit control, if you want it run in a 64-bit environment, when you compile your application, you should set the compile option to x86.

Thanks,