This version of the CTX mosaic has been replaced by the V01 release. The beta01 version will not be archived permanently but will be retained here as a reference for published work that made use of it.
The Bruce Murray Laboratory for Planetary Visualization is generating a seam-corrected and seam-mapped mosaic of Mars rendered at 5.0 m/px using Context Camera (CTX) data from the Mars Reconnaissance Orbiter (MRO).
Introduction
Below, we provide a beta version of the entire planet. This is a preliminary product that we created (1) to generate a list of orbits that will be included in the final mosaic, (2) decipher challenges involved with generating a product of this unprecedented scale, and (3) solicit feedback from the Mars science community for how best to generate the mosaic in a way that helps as diverse a group of scientists as possible.
We emphasize transparency both in how the mosaic is being generated and for users to understand where data in the mosaic come from. We have developed a Python-based pipeline that incorporates non-destructive image processing techniques that preserve all information about the original data that comprise the mosaic and map all seams. This connects the mosaic directly with its original data, ensures that blending artifacts are not mistaken for landforms, and provides instant access to the raw data that comprise the mosaic.
Global coverage of the beta01 version of the CTX mosaic. Gaps in coverage are shown as dark strips.
This product contains CTX data through MRO release 45, CTX mission phase J21 (December 1, 2017). The mosaic is comprised of 74,196 separate CTX images out of 95,106 available (78.0%). Orbits not included were either (1) low signal/noise (increased atmospheric opacity), or (2) redundant coverage (stereo targets, change detection, etc.). With low-quality images removed, the mosaic covers 97.3% of Mars.
The mosaic covers 88°S to 88°N. All data are resampled to 5.0 m/px, including 2,334 orbits (3.14%) that are natively higher resolution than 5.0 m/px, though no images are higher resolution than 4.93 m/px. The planet was divided into 3,960 4°x4° tiles, which are not blended with each other in this version. As we describe below, every pixel can be traced to its host orbit through vectorized seam-maps that are provided with every tile. These seam-maps provide pixel-for-pixel spatial documentation of image seams, to prevent misinterpretation of seams as possible landforms and to provide instant access to original data. A completely seamless CTX mosaic is currently infeasible, so seam-maps are required to prevent misinterpretation.
The entire mosaic, without overviews built, is 3.9TB as 3,960 zip files. Assuming a 10 mb/s connection, the mosaic takes ~4.5 days to download. With level-7 overviews built, the mosaic occupies 5.3TB of space. All tiffs are LZW compressed. This is a beta product and will not be archived. We cannot guarantee that the links to the beta01 release will remain active in perpetuity.
Major Imaging Parameters for CTX Data in the Mosaic
Below are plots of the principal imaging conditions for the 74,196 separate CTX images used in the Beta01 mosaic.
Where the Data Come From
Below is a broad overview of the processing pipeline that produced the CTX Global Mosaic. The pipeline consists of three stages, once a list of all images within a 4.25°x4.25° tile is generated: (1) Calibration and Projection, (2) Non-Destructive Blending, and (3) Vectorized Seam-Map Generation. Our description of the mosaic in our 2018 LPSC abstract applies to this beta01 release, so that is an appropriate reference if needed.
Calibration and Projection
Each orbit within a tile is automatically downloaded and undergoes routine ingestion from a PDS EDR .IMG file to an ISIS3 cub file, by using mroctx2isis and spiceinit. The built-in radiometric calibration ctxcal is applied, in part, to help reduce the smile artifact that reduces pixel values on the margins of the pushbroom-detector relative to the center of the detector.
This calibrated product is used to test whether that orbit is of sufficient quality to be included in the blended mosaic. High-quality images have a significantly reduced smile artifact after calibration, while this artifact is more apparent in dusty images. We calculate the degree of this artifact by analyzing the statistics of the lateral edges of the image and compare those to the center-most 30 pixels. Images with less disparity between the edges and the center are deemed sufficient for blending, while those that do not are removed from the pipeline. This has proved quite effective for the beta01 product, but it is possible that some acceptable orbits were excluded. We encourage users to use the feedback form below to alert us to acceptable data that are not included in the beta01 mosaic.
Orbits that pass our threshold test still retain some smile artifacts that can be difficult to discern by eye but significantly inhibit automated blending. Thus, we apply a column-based normalization routine using cubenorm that successfully balances tones laterally across the image. This is essential for avoiding burnt seams in the blended product, though does introduce vertical striping artifacts for low-quality images (see "Limitations..." below).
In preparation for blending, each orbit undergoes a tone-matching routine that improves the odds for seam-removal during the blending stage. The entire beta01 product used the same tone-matching algorithm, though we anticipate that for future versions we will take advantage of our modular approach and apply different equations to different tiles, depending on the nature of the terrain. For this version, we use a variable standard deviation stretch that is calculated based upon the variance of the specific orbit being processed. This is required for blending overlapping orbits with dramatically different histograms, like orbits of the walls of Valles Marineris that overlap with the featureless plains adjacent to the rim of Valles Marineris. Without this correction, orbits that only cover featureless plains become stretched too harshly relative to their greater-variance neighbors, preventing effective blending. Thus, we are sacrificing high-frequency information on featureless plains to accommodate smooth blending.
Each orbit is projected onto a 4.5°x4.5° canvas at a fixed resolution of 5.0 m/px in equidistant cylindrical projection, with fixed boundaries that extend 0.25° in all directions from the desired 4°x4° quad. This is done to allow for 0.25° of overlap on all sides so that tiles can be blended together (though that was not done for the beta01 release). Each orbit is rendered as a 53,348 px x 53,348 raster that is exported as a Tiff with LZW compression so as to suppress file sizes (given the large volume of null pixels in each raster). Once all orbits are processed to this level, they are ready for non-destructive blending.
Non-Destructive Blending
The CTX mosaic has served as an experiment at implementing non-linear, non-destructive image processing into a traditional workflow. Sequential processing (like ISIS3, GDAL, etc.) creates successive generations of files that do not necessarily retain information from the previous step. This is problematic for a project as large as the CTX mosaic, both in terms of hardware resources and for data traceability. For this reason, we explored non-destructive processing, which is standard in nearly every other professional field of image/video processing. In non-destructive processing, processes are fully reversible, and are detatched functions that can be applied at the discretion of the operator. This provides (1) radically improved efficiency and (2) full preservation of information, allowing for the generation of vectorized seam maps for the CTX mosaic. This allowed the CTX mosaic to be generated efficiently without high-performance computing resources.
After testing an array of blending options, by far the most efficient and high-quality platform was Adobe Photoshop. Photoshop is capable of blending > 200 full-resolution CTX images at once on a typical desktop workstation, greatly out-performing traditional, destructive scientific software that eventually must lean on high-performance computing support, all while preserving all information from the original orbits. Photoshop can now be operated via Python by using the comtypes module, allowing for full automation and integration with more traditional geoprocessing packages (gdal, arcpy, etc.).
Non-destructive blending is acheived by determining the path of least contrast between overlapping orbits (a seamline). For the CTX mosaic, this typically occurs in featureless plains where disparities in imaging conditions are less dramatically manifested. Thus, features of more geologic interest are more likely to be captured within one orbit, reducing artifacts. Once seamlines are generated, masks are applied to shield data that are not used in the final composition. The inverse of each orbit's mask represents the portion that is retained and is used to generate the vectorized seam map (see below). After masks are applied, contrast is adjusted radially away from the seamline so that the images match but are adjusted less and less away from the seamline.
Vectorized Seam-Map Generation
Masks that shield data for each orbit are exported as LZW-compressed rasters with values of 0 where data are used in the mosaic and 255 where data are not used. Each raster can then be vectorized by using the GDAL python script gdal_polygonize.py. This vectorizes the entire tile and attaches the DN value (0 or 255) as an attribute. This field is used to remove all DN=255 features, which represent regions not used by that specific orbit. If an orbit was not used at all during the non-destructive blending stage, it is removed entirely at this step. Features that do survive this stage have their DN attribute removed and a PRODUCT_ID field added, with the orbit number assigned in this field (derived from the filename of the raster that was vectorized).
Once all features have been vectorized, they are appended to a centralized shapefile for that tile. This produces one shapefile with all features with thier PRODUCT_ID field assigned. This field is used to apply a join to the original CUMINDEX table from the PDS, which appends all spacecraft and imaging conditions metadata to each feature. Finally, we add an attribute field called ASU_LINK and build a URL for each feature's original orbit, which finally directly connects the mosaic product to the data that were used to build it.
Limitations of the beta01 Release
The beta01 release of the global mosaic is a preliminary product and as such should be used with caution for detailed scientific analysis. Its primary purpose is to engage the Mars science and education communities on what improvements can be made to improve the quality of the mosaic and how to improve the convenience and efficiency of accessing it. We welcome all constructive suggestions through our feedback form below.
Image Seams
Due to variations in atmospheric and surface conditions on Mars, a completely seamless CTX mosaic is infeasible without severe manipulation of the data. This is particulary true at high latitudes where seasonal frost generates radically variant reflective properties depending on the season of the observation. Our effort in generating pixel-for-pixel vectorized seam maps was made so that the user can instantly decipher whether they are observing a real surface contact or an image seam.
The majority of low- and mid-latitude seams are due to variations in the signal/noise of neighboring orbits. Our pipeline automatically removes images of extremely low quality, but images of moderate quality are frequently blended with images of high quality. Without degrading the quality of the high quality orbits, we accept these seams so as to preserve as much fidelity as possible and urge users to consult the seam maps to make sure they properly interpret the data. Part of the generation of the next version of the mosaic will be replacing moderate quality orbits with new, high-quality orbits wherever possible.
Our blending technique is most effective on rugged, heavily-cratered low-latitude terrain with a lot of high-frequency information. Inversely, our blending technique is least effective on broad, featureless plains. Orbits covering these units have narrow histograms such that subtle variations in imaging conditions between neighboring orbits are amplified significantly upon blending. We are experimenting with improved tone-matching equations for these regions that improve the ultimate blended product without degrading the quality of the data unnescessarily.
Tile Blending 8°x8° sample of of the CTX mosaic showing seams between the 4°x4° tiles.
The beta01 release consists of 3,960 4°x4° tiles, each of which has orbits within them blended together. But the tiles themselves are not blended together. We have written software to do this and results from this can be seen in our regional samples below. See the sample above as an example for how tiles look when rendered together.
Image-to-Image Registration
For this first beta version, we did not register overlapping images to each other, so each image is referenced only using SPICE data. This beta version filtered out over 20,000 CTX images that are not used in the mosaic (due to either poor image quality or redundancy), so not registering them avoided unnecessary processing. Image mis-registrations are rarely noticeable, though a few examples have been found. This is due to excellent pointing of CTX relative to image resolution, and to our non-destructive blending process, which is extremely forgiving to mismatches. If precise spatial measurements need to be made that span multiple images, we encourage users to access the processed CTX images through the ASU Mars Image Explorer (each orbit is linked as an attribute in our seam-mapping shapefile provided with each tile). The next version of the mosaic will include image-to-image registration.
Vertical striping artifacts
All CTX images contain radiometric artifacts, including a "smile" feature, which manifests as all orbits having anomalously lower DN values at the ends of the detector. All images have the ISIS3 built-in calibration routine ctxcal applied, which helps to remove this effect but further correction is needed for blending. Each image has a column-based normalization function applied (cubenorm), which balances tones across the image. While this works perfectly for high signal/noise data, it can introduce vertical stripes in lower quality data.
Stray null pixels
Our tone-matching algorithm applies a variable standard deviation stretch to each image before blending, with the number of standard deviations from mean calculated based upon the variance of the image. In some instances, this sets low DN value pixels to 0, which eventually gets converted to null. Thus, some very steep, shaded slopes have null values where there should be data. This issue will be fixed in the next version of the mosaic. In ArcMap, if there are no layers visible beneath the CTX mosaic, the frame background can be set to black to fill in the null values. Likewise, in QGIS, the the project background can easily be set to black through the Project Properties.
Geodetic control
At the moment, we do not have plans to produce a geodetically controlled version of the CTX mosaic, as the time this would take would delay the release of a seamless global mosaic that satisfies the needs for almost all Mars science projects. We are developing and plan to release a version of the mosaic that is registered to the global Mars Orbiter Laser Altimeter (MOLA) dataset. The beta01 release should not be used for precise measurements of the physical location, with calculated uncertainties, of features on Mars.
Access the Data
The beta01 version of the CTX mosaic can be streamed over the web and through GIS applications as a Web Map Service (WMS) through ESRI, who have generously processed the mosaic and hosted it for us. ArcMap users (tested in ArcGIS 10.6) can load it via the website or from within ArcMap by using "Add Data From ArcGIS Online..." and searching for "Mars CTX." QGIS users can download this file that points to the data online. This version is JPEG compressed and will be sufficient for most purposes, but we encourage users to use it with our quad map shapefile to quickly access the orignal tiles if needed.
The Beta01 release can also be streamed through JMARS. JMARS is a free, java-based, user-friendly GIS from Arizona State University that integrates a wide array of remote sensing data of Mars. We greatly appreciate the work of the JMARS team for working to make the CTX mosaic easily available to the JMARS user base.
To download specific tiles, please use our new Mars Data Browser, developed by Daven Quinn. Draw a box around your region of interest, click the "CTX Global Mosaic" button, and select the tiles you want to download. This will generate a list of URLs of the zip files that contain the data with a "curl -O" prefix so they can be downloaded as a batch using Linux or MacOSX or with Cygwin on Windows.
Data are also available through a directory listing. Tiles are in subdirectories by their easternmost longitude (i.e., all tiles that cover 12°E-16°E are in the E012 directory). Tile names include the lower-left coordinate for that tile (i.e. the tile E012_N24 covers 12°-16°E and 24°-28°N).
We have also produced a grid shapefile (.zip) that shows the tile distribution across Mars. Each feature contains links to (1) the data for that quad and (2) a .png thumbnail.
Engaging with the Mosaic
Each quad's zip file contains (1) Four 2°x2° .tif GeoTiffs, (2) One .shp shapefile (with ancillary files) with seam mapping and links to original orbits, and (3) One .vrt virtual format text file that links the four GeoTiffs in one file. Overviews/Pyramids are not provided so as to decrease file transfer times. While each tile is blended as a 4°x4° tile, they are provided as 4 2°x2° GeoTiffs because some software packages cannot read the BigTIFF format. If you render the four .tif files individually in a GIS program, they will be contrast stretched individually and likely will not render properly.
Rendering Tiles
For the majority of use cases, the .vrt file will provide the easiest method for accessing the data within a GIS framework. Overviews will automatically be built for the .vrt file when added to an ArcMap or QGIS project. If working with many quads, overviews are most efficiently generated by using the GDAL gdaladdo tool. The following command will build overviews on one .vrt file:
In a bash terminal, to build overviews for all .vrt files within the current directory:
for i in *.vrt; do gdaladdo -ro --config COMPRESS_OVERVIEW DEFLATE ${i} 2 4 8 16 32 64 128 done
The equivalent can be accomplished in ArcGIS by using the Build Pyramids tool. A basic python script to accomplish this is here:
import arcpy import glob
rootdir = "C:/Path/To/Directory/With/Rasters/"
for i in glob.glob(rootdir + "*.vrt"): arcpy.BuildPyramids_management(i)
Stitching tiles back together
In some instances, like generating 3D perspective views, it may be preferable or necessary to mosaic the 2°x2° tiles together into their original 4°x4° format. We recommend using the GDAL python script gdal_merge.py to do this for all .tif files within the directory (followed by gdaladdo to build overviews):
The same can be accomplished in ArcGIS with the Mosaic To New Raster tool. In QGIS, the gdal_merge.py routine can be accessed in a GUI if the GdalTools plugin is installed. See here for a tutorial.
Rendering large volumes of the CTX Mosaic
We have found that the most efficient way to interact with many tiles of the mosaic, or the entire mosaic, is by using the Mosaic Dataset feature in ArcMap. This will generate one layer that references all tiles within the mosaic and works extremely efficiently once built.
For regional studies, tiles can be linked together using the GDAL application gdalbuildvrt. To build a virtual (.vrt) file that links all GeoTiffs within a directory:
gdalbuildvrt output.vrt *.tif
To build a virtual file of all GeoTiffs within all subdirectories:
Demo of accessing a processed orbit from ASU through the CTX Mosaic seam maps (ArcGIS 10.4).
Every pixel can be traced back to its original orbit instantaneously by using our vectorized polygon seam maps. This is important for validating observations within the mosaic and accessing either raw or less-processed versions of the data. Each polygon has been joined to all data provided for that specific orbit in the CTX CUMINDEX.TAB file provided via the PDS. Thus, all metadata for each pixel is contained in the attribute table for that tile's shapefile. In addition, we have provided a direct link to that orbit's page on ASU's Mars Image Explorer. This page contains a zoomable version of the orbit, and a pyramidized TIFF file that, once downloaded, will render quickly within your GIS project. This efficiently connects the mosaic with the data that were used to construct it.
Given the colossal scale of the CTX Mosaic, the vectorized seam maps are essential for quality control. If you encounter a low-quality image, or any other issue with the mosaic, please use the feedback form below to let us know not to include these orbits in the next version.
Regional Blended Samples
Below are sample mosaics from specific regions across Mars. Unlike the global beta01 product, these samples have tiles that are blended together. Please note that these were made at various stages through the construction of the global product, so the data/coverage in these samples may not match those of the global mosaic.
A Google Mars .kml file with all of the sites listed below can be found here. Please turn off the "Global Maps" radio button when viewing these data.
Melas Chasma
Google Earth/Mars .kml file. Make sure to turn off the "Global Maps" radio button.
Batch download scripts using wget and curl.
We are thankful for the effort of Lucian Plesea of ESRI to make the beta01 release streamable.
This project has benefited from feedback we have received from the scientists listed below who have scrutinized regional samples for us that helped produce this beta release.
All data used in the construction of the CTX global mosaic have been publicly released and are freely available via the NASA Planetary Data System. The CTX camera was built by Malin Space Science Systems, the Mars Reconnaissance Orbiter was built by the Jet Propulsion Laboratory/NASA. The Murray Lab/Caltech grants free use of the beta01 version of the mosaic for all purposes. Data credit should be to: NASA/JPL/MSSS/The Murray Lab.
