Just a few days before I was going to take a course in python scripting, I noticed the command line button on the ESRI toolbars.

It is there in ArcMap and ArcCatalog. I wonder why I’d never noticed it before. Turns out you can do a lot of cool things with it. The portion of every Help document labeled “Command line syntax” is talking about things you can type into the command window. Plus, I learned in my class later that with ArcGIS 10, it will be renamed to the Python Window because we will be able to run python scripts directly from it! But I am getting ahead of myself.

There are certain things you can only do from the command window, and today I will talk about one of them: converting metadata from one format to another. The two prominent metadata standards are FGDC: Federal Geographic Data Committee and ISO: International Organization for Standardization 19115.

If you want to use data from an organization that follows a different standard than you do, you might need to do a conversion. For example, the metadata for World Wildlife Fund’s Global Lakes and Wetlands Database is in FGDC format. In order to view it in ArcCatalog, I need to have an FGDC stylesheet selected.

With the right stylesheet, all of the pertinent information is displayed in the metadata window.

If I switch to an ISO stylesheet, what I see is not nearly as helpful.

You could consign yourself to switching between stylesheets, but I suspect many people work in places that insist all of the metadata for their collection of geospatial records be in the same format. If you need to do a conversion, there is a faster way than typing all of the information back into the metadata creation wizard.

ESRI has provided a set of four metadata translators with the ArcGIS desktop install. They should be located in the \Program Files\ArcGIS\Metadata\Translator folder. You can run these translators from the command line window. First, pick the one you need.

• FGDC to ISO 19139 (FGDC2ISO19139.xml)
• ESRI-ISO to ISO 19139 (ESRI_ISO2ISO19139.xml)
• FGDC to ESRI-ISO (FGDC2ESRI_ISO.xml)
• ISO 19139 to ESRI-ISO (ISO19139_2ESRI_ISO.xml)

Then press the command line button, and type ESRITranslator into the window. If you press the spacebar, a tool tip will pop up that tells you the arguments you need to enter next. In this case it will say

ESRITranslator <source> <translator> {output} {logfile}

Input within carrots <> is required; input within brackets {} is optional. For me, <source> is the path to the Global Lakes and Wetlands metadata .xml file. <translator> is the path to the ESRI Translator I want to use ( FGDC to ESRI-ISO ). And {output} is the path to where I want to store the new converted .xml file. I’m not going to specify a logfile.

My code looks like this:

(click on image to see full size)

Notice a couple things here. My cursor is in the path to the translator file, and <translator> is bolded in the tool tip. This is intentional. If I move the cursor to other arguments, the bolding will change accordingly. Use this to make sure your syntax is correct.

Also notice that I have surrounded the path to the translator file in quotes. Quotes are optional, but you must use them if you have spaces in your path name. Otherwise the space will be misinterpreted as the beginning of the next argument. Going through and checking what is bolded where will help to catch this sort of thing.

Press enter to run the code. Then, press the Import Metadata button and browse to your new .xml file.

Now it looks how ISO should!

Some spatial data will store height information in a normal attribute table field, but others will store it in the feature geometry as a z-value. Using z-values increases the feature geometry from 2D (x and y) to 3D (x,y and z). ArcScene will automatically display 3D data using the z-values, and it will do so more rapidly than draping the data over a surface.

Because z-values are optional, you may run into a situation where you need to merge a layer that does not have them into a layer that does. For example, say you have a geodatabase holding tree locations along neighborhood streets. The height of the trees is stored in z. Your surveyors have gone out and collected the locations of some additional trees in people’s yards, but they didn’t use 3D geometry. You need to load the new yard trees into the street tree feature class.

Right clicking and selecting properties in ArcCatalog confirms that the StreetTreePntZ feature class has z-values but the YardTree shapefile does not.

Trying to load the shapefile as-is will result in the error message.

One solution: you could run ET GeoWizards Point to PointZ conversion on the shapefile.

Another solution: Environment Settings! Import the yard tree shapefile into your geodatabase as a separate feature class. But before you press the OK button, press Environments… right next to it.

Expand General Settings and under Output has Z-Values, change Same as Input (the default) to Enabled.

Now you will be able to load the z-enabled yard tree feature class into the StreetTreePntZ feature class.

The z-values for your newly loaded features will be 0. You can edit z values in ArcMap by starting an edit session, double clicking on each feature (to turn it into a sketch) and then opening the Sketch Properties dialog from the Editor toolbar.

In the future I will write an entry about visualizing these cool 3D features in ArcScene.

This issue most often arises for me when I am in ArcCatalog using the simple data loader tool. It can also arise when trying to import a feature class into a feature dataset, when creating new features during an edit session in ArcMap, or when performing spatial adjustment to move features to a new location. In each case, the error occurs when the extent of the feature class or feature database is not large enough to contain the new or moved features. The solution would appear to be “increase the extent,” however, it is not possible to change the extent once a feature class or feature dataset has been created. You need instead to create a new one with the correct extent. I will show you how to do that, returning to the example of the Oceanic layers I used in my post about preparing layers for merging with model builder.

Today, I would like to merge all my city point shapefiles into one feature class in my master Oceanic geodatabase. I have city points for Australia and Papua New Guinea. If I merely import the Australia shapefile as a feature class, and then attempt to load the Papua New Guinea points into that feature class, I will get this error message:

One of the points is out of bounds! By displaying the layers in ArcMap, it’s easy to see what happened. When I imported the Australia shapefile, the spatial domain was defined as the extent of that layer. The one Papua New Guinea point that didn’t get loaded fell too far east of that domain.

A good rule of thumb is: if you have a dataset that covers your entire domain, import that first and then load the other datasets into it. However, you often won’t. In that case you are going to have to take one or two extra steps.

Single step method (good enough for most purposes):
Create a new feature class. ArcCatalog will ask you to chose the projection of the feature class, and when you do, the extent will be set to the all of the mathematically possible values in that coordinate system. In other worlds, the whole world (and then some). ArcCatalog will then allow you to chose a schema for the new feature class. Import the schema from one of the layers you are going to load in.

You will now be able to load all of your layers into the new feature class without any out of bounds errors.

Two step method (better):
Add all of your layers to ArcMap. Then draw a box around all of the features and save it as a separate polygon shapefile.

This polygon shapefile will be used to precisely set your x,y domain. In order for this to work, the projection of the layer must be set to unknown. Clear the projection if it has one defined. Now, create a new feature class in ArcCatalog. When it asks you for the projection of your new feature class, import the projection of your polygon shapefile.

If this polygon has a defined projection, ArcCatalog will just do what it did before and set the extent as all mathematically possible values in that coordinate system. But if you import an undefined projection, ArcCatalog sets the domain as the spatial extent of that dataset. That’s exactly what you want. You’ll see an additional screen that shows you this is happening and even allows you to change the x,y domain values if you want.

Here, you can see what the domain looks like if it is defined automatically by the GCS WGS84 coordinate system.

And here is what is looks like when defined by my domain polygon with an unknown coordinate system.

Why is the second way better? It is preferable to have as small a domain as possible because a smaller domain allows for a greater resolution. Resolution is the minimum distance between coordinates. A higher resolution means that locations can be placed more precisely, because their coordinate positions are allowed to have more decimal places. Because data storage is finite, there will always be a trade off between extent and resolution. In most cases, the resolution gain will not be noticeable, especially if you are using a post-9.2 high precision geodatabase. But I put it here as a best practice.

Once you have loaded all your layers into your new feature class (error free!), define its coordinate system as your last step. This feature class is totally optimized for the data it’s holding.

Most of the time, ArcMap “just works” invisibly behind the scenes to line up everything correctly. It doesn’t matter if the datasets are in different coordinate systems, and sometimes it doesn’t even matter if the coordinate systems are defined. This can make the times when something goes wrong seem like mysterious aberrations, hard to troubleshoot. But there really are only two reasons why it works and why it doesn’t.

Datasets will line up correctly if

1. They are in the same coordinate system, even if one or more datasets is <undefined>
2. They are in different coordinate systems, but all coordinate systems are correctly defined. ArcMap will do on-the-fly projection.

Datasets will not line up correctly if

1. They are in different coordinate systems and one or more datasets are <undefined>
2. One or more coordinate systems are incorrectly defined

If you are suffering from problem #1 or #2, the fix is running ArcToolbox’s “Define Projection” on any undefined or incorrectly defined datasets. That is, if you are working with typical ArcMap data like shapefiles and feature classes. The fix is different for AutoCAD data.

The City of Bloomington, Indiana, offers their GIS data in both shape and dxf format, so we can experiment. In this example, I have projected a road shapefile into GCS WGS84, but I have left the corresponding dxf road layer alone. The dxf is now in a different coordinate system than the shapefile, and that coordinate system is not defined. Hence, the two layers which should be exact duplicates are showing up worlds apart.

They will be brought back together if ArcMap knows what the coordinate system of the dxf is. But, as I said above, this shouldn’t be done like normal with ArcToolbox. Instead, it is better done in ArcCatalog. If you right click on the roads_city.dxf layer in ArcCatalog to bring up its Properties, you will see that the coordinate system is Unknown. Pressing the Edit… button will allow you to select a coordinate system. The metadata on the City of Bloomington website says this data is in NAD 1983 StatePlane Indiana West FIPS 1302.

Note: Be sure to edit the projection on roads_city.dxf itself and not the individual point, polyline, polygon components. The new projection will carry over into the entire collection.

Selecting this coordinate system will prompt you to save a .prj file. Be sure it is in the same directory with the same name as the dxf.

This is what is inside the .prj file if you look at it with Notepad:

PROJCS["NAD_1983_StatePlane_Indiana_West_FIPS_1302",GEOGCS["GCS_North_American_1983",
DATUM["D_North_American_1983",SPHEROID["GRS_1980",6378137.0,298.257222101]],
PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Transverse_Mercator"],
PARAMETER["False_Easting",900000.0],PARAMETER["False_Northing",250000.0],
PARAMETER["Central_Meridian",-87.08333333333333],PARAMETER["Scale_Factor",0.9999666666666667],
PARAMETER["Latitude_Of_Origin",37.5],UNIT["Meter",1.0]]

Note: Shapefiles have these too! If you delete them, the projection goes to Unknown. For some reason, ArcToolbox’s “Define Projection” tool creates these .prj files for shapefiles, but it doesn’t do so reliably for AutoCAD data. In some instances it only creates an .xml file for the AutoCAD data, which isn’t enough to tell ArcMap the projection. This problem appears to be fixed with 9.3, but 9.2 users will have more luck the ArcCatalog way.

All users will have better luck getting projection assignments to “take” if they make sure that:

1. The dataset is not concurrently open in ArcMap
2. The folder that holds the dataset is not set to “read-only”
3. There are no spaces in the file name or path

Now, if you are successful, when you add the road dxf to an ArcMap session with your road shapefile, you will get a message asking you what geographic transformation to use. This means ArcMap has recognized the projection.

Pick a transformation and you’re good to go.

Now, what do you do if you don’t have metadata like the good City of Bloomington provides, so you don’t know the coordinate system of your CAD data? You have two options:

The easy way: Georeference it just like an image! This ability was added in ArcGIS 9.2. You can see below that I’m able to select the dxf in the georeferencing toolbar.

There’s some minor differences, so read here if you want the details or if you’ve never learned to georeference images:

• Transforming CAD datasets (ESRI Webhelp)

The hard way: If you’ve got an older version than 9.2, you will have to create a world file from coordinates at the corners. It’s an arduous process but it will work. Instructions here:

• HowTo: Transform CAD data to line up with other data (ESRI Technical Article)

I have done this successfully with CAD maps that have rectangle frames around them, making it easy to find the corners. I can see it being very difficult otherwise. If the world file will not work for you, there’s one other option.

The other hard way: Spatial Adjustment. In other words: “Georefrencing for vectors.” It works for shapefiles without attainable coordinate systems as well. I will be covering it in a later entry.

This is a comment data management task. You may receive layers that are unnecessarily split into a lot of small files. For instance, I often see roads divided into separate layers for each road type. Highway, major, minor, trail, etc. — all in their own shapefile. It is useful to be able to distinguish between road types on a map, but it is better done through the use of symbols or definition queries than by having an individual layer for each.

By the same token, themes are often split up by region. You might have all the road types together, but in a separate shapefile for each county, state or country. You might rather have a single worldwide or countrywide coverage. But as with the road types, it is a good idea to maintain the ability to distinguish between region, in case the need should arise.

The way to do this is to add a new attribute field, populate it with the characteristic you are trying to preserve, and then merge. In other words, you’re moving the distinguishing characteristic from the file name or file folder to the attribute table, so it is not lost, prior to merging.

For example, I have a list of basemap layers for the Oceanic countries of Australia, Paupa New Guinea, New Zealand and Samoa. I want to merge all of the like layers into one for the entire region, so I have an Oceanic road layer, an Oceanic river layer, etc. But before I do that, I want to add a country field and put the country name in there. This way if I need to select out just the Australian roads later, I can.

It’s pretty tedious to open each attribute table, add a field, and then run the field calculator to stick the country name in there. This task can be sped up considerably through the use of model builder. Model builder allows you to batch process ArcToolbox commands, and to chain commands so that the output of one tool into feeds into the input of the next tool. We will be able to add new “Country” fields to all our Australia layers at once, and immediately insert the value “Australia” into those fields, with a single click of mouse.

Let’s get started. Open up ArcCatalog and activate ArcToolbox. If you have never used model builder before, you will need to create a new toolbox to hold your model. Then, create a new model within that toolbox.

Now, drag the tools that you want to involve into the model screen. In this case, that’s “Add Field” and “Calculate Field.”

We’ll be starting with Add Field. It’s time to choose your input layer. By default, the tools dragged into model builder accept a single data layer as input, but you can change them to accept a list of layers (batch mode). To do this, right click on the Add Field tool to make the Input Table a variable. Setting something as a variable “exposes” it, which means it makes it editable independent of the rest of the settings that you will configure later. Model builder indicates a variable has been exposed by giving it its own symbol (an oval) in the model flowchart. The other settings remain hidden inside the tool symbol (rectangle).

Now, right click on the Input Table variable to bring up its Properties. Then, change it to a list of values.

The Input Table symbol will now change to a stack of ovals. If you double click on the stack, you will be able to cue up as many inputs as you like. The fastest way to generate a list of inputs is to drag them from windows explorer. Note: You can’t drag them from ArcCatalog because the input box maintains focus until you dismiss it.

You can navigate to the folder, and manually select all the .shp files. Or, even faster, you can do a file search for them. This gets any subfolders too! I’ve searched within my Australia folder.

If it works right, your stack of ovals will turn blue. Now it is time to set the rest of the Add Field settings. You can access them by double clicking on the Add Field rectangle. Here, you will see your list of settings. The Input Table column is filled in but the rest of the settings (field name, field type, etc) are grayed out, except the first row. Double click on that first row to edit the other settings. Here, input your field name (“Country”), field type (“TEXT”) and field length (“50″) just like you would when activating the tool from ArcToolbox. Note: If you leave the text field length blank, it defaults to 50.

Because you don’t have any of these parameters set as variables, the settings you enter here will propagate down through the entire list of inputs. When you finish, Add Field should be yellow and the output oval stack should be green. This indicates all of the settings are valid, and they are ready to run.

Next, configure the settings for the Calculate Field tool. You’ll want to set the input of this tool as the output of the Add Field tool. Just look at the name on the oval extending outwards from Add Field. (The name is generated from the first layer in your input list). In my case, the output of Add Field is called “admin_area (3)”. I’ve set that as the Input Table for Calculate Field. Then, use the dropdown list to select the Field Name you want to populate. Your new field, “Country”, will show up in the list of options because model builder runs the commands in order and knows you’re planning to add that field in the previous step. Finally, type the string you would like to use to populate the field in the Expression box, with quotes around it. In this case, “Australia.”

Now, you can run your entire model from the model menu, or one tool at a time by right clicking it. I like to do one tool at a time at first so I can be confident each step is working correctly. The model parameters will turn red while running, and will form drop shadows once they have finished.

Now, you can rerun the same model for the other 3 countries. All you need to change is the initial (blue) input list and the Expression string within Calculate Field. If you want to do even less babysitting, you can make the Expression a variable and do all four countries at once. And of course, if you want to add and calculate more fields, you’d just string them along down the chain.

Remember any ArcToolbox tool can be dragged onto Model Builder, so the potential uses are limited only by your imagination. For more information, see (both PFDs):

• Geoprocessing in ArcGIS (ESRI Tutorial)
• Model Builder: An Introduction (ESRI Conference Presentation)

In the world of shapefiles, it was relatively easy to do this. You could open up Windows Explorer and add up the sizes of the component shapefile parts:

.dbf = attribute table
.shp = feature geometry
.shx, .sbn, .sbx = various spatial indicies
.prj = projection (optional)
.xml = metadata (optional)

When we migrated into personal geodatabases, you could at least see the size of the entire database by looking at the Microsoft Access .mdb file. You couldn’t drill down into individual feature classes, however. Then, when the file geodatabase came along, things got even worse. The characteristic feature of this data type is storage as file folders. And yes, the file geodatabase looks like a folder in Windows Explorer. But the contents of that folder are unintelligible.

The image below shows how these three most commonly used data types appear in Windows Explorer:

A peek inside the FileGeodatabase folder:

ArcCatalog, the file navigation system specifically for GIS data, is clearly the superior option here. However, just like with Windows Explorer (where you have to switch to detail view to see more than a list of thumbnails) it needs some tweaking before you can get the information you want out of it.

Here’s how: Go to the Tools menu and select Options, then the Contents tab. This will give you a list of the details that show up on the Contents screen when you are navigating through your file system. Name and Type are the only ones checked on by default. You will at least want to add Size and Modified.

Note: Date modified is useful to display for so many reasons, not the least of which are

• Gives a ballpark idea how current the dataset is, if you don’t have that information elsewhere
• Helps you differentiate between different versions that may be named the same thing
• Sorting by date modified is a one click way to gather together everything you were last working on

While you’re there, take a look at what else is available. The settings you select here will remain every time you open up ArcCatalog. Ever since I stumbled upon this, it is one of the first things I do when moving to a new machine.

I’ve taken ESRI’s USA major roads data and saved it into a shapefile, personal geodatabase, and file geodatabase. With these new settings, here’s how they appear in ArcCatalog’s Contents screen.

This is a good place to point out the advantages of using file geodatabases. As you can see, they use dramatically less space to store the same amount of data. This means faster load times, as well. The other two won’t be going away any time soon: Shapefiles are widely used and good in a pinch, and personal geodatabases are nice if you like using Access to edit attribute tables. But for long term file storage, file geodatabases really are the way to go. As a bonus, you can compress them as read-only, which saves even more space.