Thursday, 27 December 2018

Google Earth Engine - Error downloading cloudfree Sentinel image


I got the following error message after trying to import cloud-free Sentinel-2 imagery for a polygon area covering the entire Tiwi Islands, Australia.




true-colour image: Layer error: internal error



// Filter to only include images intersecting Tiwi.

var polygon = ee.Geometry.Polygon({
coords: [[
[129.90415998438357,-12.027354108834277], [131.63999982813357,-12.027354108834277],
[131.63999982813357,-11.101721188177093], [129.90415998438357,-11.101721188177093],
[129.90415998438357,-12.027354108834277]]],

geodesic: false
});

// Define the image collection we are working with by writing this command
var image = ee.Image(sent2

// filter to get only images in the date range we are interested in
.filterDate("2018-09-01", "2018-11-30")

// sort the collection by a metadata property, in our case cloud cover is a very useful one

.sort("CLOUD_COVERAGE_ASSESSMENT")

// select the first image out of this collection - i.e. the most cloud free image in the date range
.first());

// print the image to the console.
print("A Sentinel-2 scene:", image);

// Define visualization parameters in a JavaScript dictionary for true colour rendering. Bands 4,3 and 2 needed for RGB.
var trueColour = {

bands: ["B4", "B3", "B2"],
min: 0,
max: 3000
};

// Add the image to the map, using the visualization parameters.
Map.addLayer(image, trueColour, "true-colour image");


python - Getting time dimension from GRIB file?


I have a GRIB file from NOAA that contains several variables for a specific subregion in Norht-East Atlantic.


I'm using ECMWF grip-api Python library to parse the file. While I'm able to get the values for each {latitude, longitude} pair for each GRIB message with this code:


def iterate_latitude_longitude():
f = open(INPUT)


while 1:
gid = grib_new_from_file(f)
if gid is None: break

iterid = grib_iterator_new(gid,0)

#Get the missingValue for this GRIB message (e.g. NaN)
missingValue = grib_get_double(gid,"missingValue")

i=0

while 1:
#Get the value from the geo iterator obtained previously, value is a tuple with [latitude, longitude, value]
result = grib_iterator_next(iterid)
if not result: break
[lat,lon,value] = result

sys.stdout.write("-Point # %d - lat=%.6f lon=%.6f value=" % (i,lat,lon))
#Check if missing value is present or not
if value == missingValue:
print "missing"

else:
print "%.6f" % value
#increase iterator
i += 1

grib_iterator_delete(iterid)
grib_release(gid)

f.close()


While this is great, this file contains forecast data, it was extracted from this place, more precisely, this file:


multi_1.nww3.t00z.grib2


Which contains world data. There are other files for different times: 00-60-12-18-00, thus, every 6 hours.


What I'm missing is the relation between the value obtained for each latitude,longitude and the time dimension. Does anybody know how to extract this info from a grib file, whether using Python or Java?


I know I can use things like wgrib2 or pygrib to convert to CSV and do the other way around...but it just looks so inefficient



Answer



You probably didn't want to just iterate over the whole file, but rather iterate over each of the (interesting) values, for each of the time steps you want.


Here is some code adapted from the sample code


import traceback
import sys


from gribapi import *

INPUT='multi_1.nww3.t00z.grib2'
VERBOSE=1 # verbose error reporting

def example():
f = open(INPUT)

keys = [

'stepRange',
'shortName',
]

while 1:
gid = grib_new_from_file(f)
if gid is None: break

for key in keys:
if not grib_is_defined(gid, key): raise Exception("Key was not defined")

print '%s=%s' % (key, grib_get(gid, key))

missingValue = grib_get_double(gid,"missingValue")

iterid = grib_iterator_new(gid,0)
i=0
while 1:
result = grib_iterator_next(iterid)
if not result: break


[lat,lon,value] = result

sys.stdout.write("- %d - lat=%.6f lon=%.6f value=" % (i,lat,lon))

if value == missingValue:
print "missing"
else:
print "%.6f" % value

i += 1


grib_iterator_delete(iterid)
grib_release(gid)

f.close()

def main():
try:
example()
except GribInternalError,err:

if VERBOSE:
traceback.print_exc(file=sys.stderr)
else:
print >>sys.stderr,err.msg

return 1

if __name__ == "__main__":
sys.exit(main())


And some sample output:


stepRange=0
shortName=unknown
....
- 288 - lat=77.000000 lon=0.000000 value=4.890000
- 289 - lat=77.000000 lon=1.250000 value=5.720000
- 290 - lat=77.000000 lon=2.500000 value=6.690000
- 291 - lat=77.000000 lon=3.750000 value=7.620000
- 292 - lat=77.000000 lon=5.000000 value=8.350000
- 293 - lat=77.000000 lon=6.250000 value=8.810000

- 294 - lat=77.000000 lon=7.500000 value=9.090000
- 295 - lat=77.000000 lon=8.750000 value=9.250000
....
stepRange=3
shortName=ws
....
- 288 - lat=77.000000 lon=0.000000 value=34.030000
- 289 - lat=77.000000 lon=1.250000 value=20.160000
- 290 - lat=77.000000 lon=2.500000 value=4.290000
- 291 - lat=77.000000 lon=3.750000 value=351.300000

- 292 - lat=77.000000 lon=5.000000 value=342.490000
- 293 - lat=77.000000 lon=6.250000 value=337.310000
- 294 - lat=77.000000 lon=7.500000 value=334.990000
- 295 - lat=77.000000 lon=8.750000 value=334.460000
- 296 - lat=77.000000 lon=10.000000 value=334.800000
- 297 - lat=77.000000 lon=11.250001 value=335.990000
- 298 - lat=77.000000 lon=12.500001 value=337.970000
- 299 - lat=77.000000 lon=13.750001 value=338.940000

Note that per your comments on the other answer, the results are in 3 hour blocks (not 6 hour as noted in the question).



normalize - Is the ground level cross section of a normalised lidar dataset flat?




If I normalise a lidar dataset I am (according to my thinking) simply subtracting the first return heights from the ground heights. Thus the resulting dataset will only contains the height above ground and a mountainous region would then appear flat when visualised as a cross section profile. Is this correct?




Wednesday, 26 December 2018

How to increase FUSION-LTK lidar processing productivity?


I am processing Lidar data with FUSION-LTK. My study area contains 5 blocks of 1000 las tiles per block. Each block is broken into approximately 4 subblocks of 250 las tiles. I am calculating grid metrics and strata metrics to assess vegetation structure. I wrote simple batch files to process entire blocks (e.g. block 1 subblock1; block 1 subblock2;...) using the command line tools in FUSION-LTK.



The problem I am encountering is that I am only able to process one subblock at a time. When I try to simultaneously start processing on another block, the current session of FUSION-LTK stops the processing and moves to the new block. How can I improve the processing workflow so that more than one subblock are processed simultaneously, or at least enhance the multiprocessing capabilities of LTK?




openstreetmap - Using OpenLayers maps with SSL


I am using OpenLayers map by using the hosted JavaScript:



 

But my client has SSL installed and when I try to run my map page it shows:


(2)[blocked] The page at https://domain.com/rwd/ ran insecure content 
from http://openlayers.org/api/2.13.1/OpenLayers.js.

So I tried https one and it turns out openlayers doesn't have one


https://openlayers.org/api/2.13.1/OpenLayers.js

Then I download the Openstreet js and hosted in client server, but then all the styles and related images are lost. Although it does show the map, basically numerous warnings pop-up in the console and I'm afraid this might get rejected at play store or so. I'm developing a hybrid application which runs on server as well.



Warnings now :


The page at https://domain.com/rwd/#/customer-plot/234 displayed insecure content from http://b.tile.openstreetmap.org/14/8743/5624.png.
The page at https://domain.com/rwd/#/customer-plot/234 displayed insecure content from http://b.tile.openstreetmap.org/14/8742/5624.png.
The page at https://domain.com/rwd/#/customer-plot/234 displayed insecure content from http://c.tile.openstreetmap.org/14/8743/5623.png.
The page at https://domain.com/rwd/#/customer-plot/234 displayed insecure content from http://b.tile.openstreetmap.org/14/8743/5625.png.
The page at https://domain.com/rwd/#/customer-plot/234 displayed insecure content from http://c.tile.openstreetmap.org/14/8744/5624.png.
The page at https://domain.com/rwd/#/customer-plot/234 displayed insecure content from http://a.tile.openstreetmap.org/14/8742/5623.png.
The page at https://domain.com/rwd/#/customer-plot/234 displayed insecure content from http://c.tile.openstreetmap.org/14/8742/5625.png.
The page at https://domain.com/rwd/#/customer-plot/234 displayed insecure content from http://a.tile.openstreetmap.org/14/8744/5623.png.
The page at https://domain.com/rwd/#/customer-plot/234 displayed insecure content from http://a.tile.openstreetmap.org/14/8744/5625.png.

The page at https://domain.com/rwd/#/customer-plot/234 displayed insecure content from http://a.tile.openstreetmap.org/14/8743/5622.png.
The page at https://domain.com/rwd/#/customer-plot/234 displayed insecure content from http://c.tile.openstreetmap.org/14/8742/5622.png.
The page at https://domain.com/rwd/#/customer-plot/234 displayed insecure content from http://c.tile.openstreetmap.org/14/8745/5624.png.
The page at https://domain.com/rwd/#/customer-plot/234 displayed insecure content from http://b.tile.openstreetmap.org/14/8744/5622.png.
The page at https://domain.com/rwd/#/customer-plot/234 displayed insecure content from http://b.tile.openstreetmap.org/14/8745/5623.png.
The page at https://domain.com/rwd/#/customer-plot/234 displayed insecure content from http://c.tile.openstreetmap.org/14/8745/5625.png.
The page at https://domain.com/rwd/#/customer-plot/234 displayed insecure content from http://c.tile.openstreetmap.org/14/8745/5622.png.

GET https://domain.com/rwd/js/lib/theme/default/style.css 404 (Not Found)
/*This the corresponding stylesheet that is loaded via Openstreet.js*/




I tried with cdn with SSL and still same issue:


https://cdnjs.cloudflare.com/ajax/libs/openlayers/2.11/OpenLayers.js



I downloaded the entire GitHub repo for Openstreet thinking this will solve the image and CSS issue.


https://github.com/openlayers/openlayers

While the above solved the CSS issues, the map however is shown by loading out images from another external site tile.openstreet.com. Seems like I have to dig into openstreet js as well..:(..





Important : I am using backbone.js.


Note : I have gone through OpenLayers 2.12 and http basic authentication woes and that has not helped me. I do not have any control over the server configuration. I just have access to a folder where I need to set up the website an everything else works fine, but this SSL is troublesome.


Changed real website address to avoid Google linking it.




open source gis - Converting tax map polygons from Shapefile to table of map number and corner coordinates


Fairfax County, Virginia provides a Shapefile here which shows the boundaries of the tax maps as polygons. Each polygon simply consists of four corners, and what I'd like to do is create a table showing the map number and the coordinates of the corners, such as the following:


Map number, bottom-left, bottom-right, top-right, top-left
77-1, -77 32, -76 32, -76 33, -77 33


(though they aren't quite square like this)


I don't have access to ArcGIS software, but do have QGIS and OpenJUMP, and can install any other software that is freely available. I believe this should be very easy but I'm just missing something. Thanks!




Answer



In QGIS you can use "Extract nodes" to first get the corner points of your polygons. Then, using "Export/Add geometry columns" you can add x and y coordinates to those points. The results will be four lines of data though.


Another way is to copy paste the geometries from QGIS' map window into a text editor. You'll get the WKT representation of the polygons, e.g.


wkt_geom
POLYGON((-0.971664 0.453443,-0.516714 0.827044,-0.245552 0.432353,-1.001793 -0.224462,-0.971664 0.453443))

performance - Understanding why arcpy.CreateScratchName() gets slower with more file geodatabases in folder?


In my ArcGIS Pro 1.4.1 application, I noticed that a short section of ArcPy code that creates a scratch name, and then uses that to create a file geodatabase (always in the same folder with the same root name), goes from taking a few seconds the first time it is run to taking 2 minutes around the 30th time.


I have isolated that it is CreateScratchname rather than CreateFileGDB that is slowing down by running this code:


import arcpy,time,os

workingFolder = r"C:\Temp\TestFolder"
for i in range(100):
scratchName = arcpy.CreateScratchName("Working",".gdb","Workspace",workingFolder)
baseName = os.path.basename(scratchName.split(".")[0])
workingGDB = r"{0}\{1}.gdb".format(workingFolder,baseName)

start = time.clock()
arcpy.CreateFileGDB_management(workingFolder,baseName)
elapsed = (time.clock() - start)
print("Creating working geodatabase {0} took {1} seconds".format(workingGDB,int(round(elapsed))))

to create this output where you will see that the time for CreateFileGDB is consistently 1 second whereas the time for CreateScratchName is 3 seconds for the first time, then climbs from 1 second to 4-5 seconds by the 25th iteration:


Python 3.5.2 |Continuum Analytics, Inc.| (default, Jul  5 2016, 11:41:13) [MSC v.1900 64 bit (AMD64)] on win32
Type "copyright", "credits" or "license()" for more information.
>>>
======== RESTART: C:\CompilationPlots\Scripts\CreateWorkingGDBtest.py ========

Creating scratchname C:\Temp\TestFolder\Working0.gdb took 4 seconds
Creating working geodatabase C:\Temp\TestFolder\Working0.gdb took 3 seconds
Creating scratchname C:\Temp\TestFolder\Working1.gdb took 0 seconds
Creating working geodatabase C:\Temp\TestFolder\Working1.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working2.gdb took 0 seconds
Creating working geodatabase C:\Temp\TestFolder\Working2.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working3.gdb took 1 seconds
Creating working geodatabase C:\Temp\TestFolder\Working3.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working4.gdb took 1 seconds
Creating working geodatabase C:\Temp\TestFolder\Working4.gdb took 1 seconds

Creating scratchname C:\Temp\TestFolder\Working5.gdb took 1 seconds
Creating working geodatabase C:\Temp\TestFolder\Working5.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working6.gdb took 1 seconds
Creating working geodatabase C:\Temp\TestFolder\Working6.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working7.gdb took 1 seconds
Creating working geodatabase C:\Temp\TestFolder\Working7.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working8.gdb took 1 seconds
Creating working geodatabase C:\Temp\TestFolder\Working8.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working9.gdb took 2 seconds
Creating working geodatabase C:\Temp\TestFolder\Working9.gdb took 1 seconds

Creating scratchname C:\Temp\TestFolder\Working10.gdb took 2 seconds
Creating working geodatabase C:\Temp\TestFolder\Working10.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working11.gdb took 3 seconds
Creating working geodatabase C:\Temp\TestFolder\Working11.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working12.gdb took 2 seconds
Creating working geodatabase C:\Temp\TestFolder\Working12.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working13.gdb took 2 seconds
Creating working geodatabase C:\Temp\TestFolder\Working13.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working14.gdb took 2 seconds
Creating working geodatabase C:\Temp\TestFolder\Working14.gdb took 1 seconds

Creating scratchname C:\Temp\TestFolder\Working15.gdb took 2 seconds
Creating working geodatabase C:\Temp\TestFolder\Working15.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working16.gdb took 3 seconds
Creating working geodatabase C:\Temp\TestFolder\Working16.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working17.gdb took 3 seconds
Creating working geodatabase C:\Temp\TestFolder\Working17.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working18.gdb took 3 seconds
Creating working geodatabase C:\Temp\TestFolder\Working18.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working19.gdb took 3 seconds
Creating working geodatabase C:\Temp\TestFolder\Working19.gdb took 1 seconds

Creating scratchname C:\Temp\TestFolder\Working20.gdb took 3 seconds
Creating working geodatabase C:\Temp\TestFolder\Working20.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working21.gdb took 3 seconds
Creating working geodatabase C:\Temp\TestFolder\Working21.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working22.gdb took 4 seconds
Creating working geodatabase C:\Temp\TestFolder\Working22.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working23.gdb took 5 seconds
Creating working geodatabase C:\Temp\TestFolder\Working23.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working24.gdb took 4 seconds
Creating working geodatabase C:\Temp\TestFolder\Working24.gdb took 1 seconds

>>>

The above would not be such a concern except that in my application I go on to fill the working geodatabase with about 50 feature classes and tables at each iteration, and this seems to make CreateScratchName run even slower.


Is the slowing down of CreateScratchName demonstrated in my code above, and anecdotally exacerbated by the size of each file geodatabase once they have been filled, to be expected?




arcpy - Changing output name when exporting data driven pages to JPG?

Is there a way to save the output JPG, changing the output file name to the page name, instead of page number? I mean changing the script fo...