Welcome to Planet OSGeo

May 16, 2022

Contributions à QGIS LTR 3.22

La dernière version maintenue à long terme (LTR) de QGIS est la version 3.22. La première version corrective de 3.22 ayant ce statut est la version 3.22.4 publiée fin février 2022.

Cette version est importante pour les utilisateurs et la communauté de développeurs car elle sera maintenue jusqu'en février 2023. Elle a aussi été testée pendant 3 mois avant d'obtenir le statut LTR qui est un gage de stabilité et de qualité.

Voici la liste des contributions de 3Liz pour cette version 3.22 de QGIS (incluant 3.18 et 3.20)

Améliorations du module Traitements de QGIS

Améliorations de QGIS Server

Amélioration de l'option Projet de confiance

Correction de QGIS Server

Nouvelles fonctionnalités globales

API : nouvelles classes

Tests, pas de fonctionnalités, mais pour être sûr qu'il n'y aura pas de régression à l'avenir

Amélioration du côté du dépôt GitHub

  • Template/modèle pour les ticket des utilisateurs
  • Notifications de release GitHub

La plupart des fonctionnalités ont été financées par l'Ifremer, ou par certains clients utilisant notre solution d'hébergement Lizmap et ayant un problème avec QGIS Server, ou sur notre temps Open Source à 3Liz.

by René-Luc DHONT at May 16, 2022 07:00 AM

Contributions to QGIS LTR 3.22

The latest long-term released (LTR) version of QGIS is 3.22. The first bugfix release of 3.22 with this status is version 3.22.4 released in late February 2022.

This version is important for users and the developer community as it will be maintained until February 2023. It was also tested for 3 months before obtaining the LTR status which is a guarantee of stability and quality.

Here is the list of 3Liz contributions for this QGIS 3.22 release (including 3.18 and 3.20)

QGIS Processing enhancements

QGIS Server enhancements

Improvement of the Trust Project option

QGIS Server fixes

New global features

API: new classes

Tests, no features, but to make sure there will be no regression in the future

Improvements on the GitHUb repository side

  • Template for user tickets
  • GitHub release notifications

Most of the features were funded by Ifremer, or by some customers using our Lizmap hosting solution and having a problem with QGIS Server, or on our Open Source time at 3Liz.

by René-Luc DHONT at May 16, 2022 07:00 AM

May 15, 2022

I stopped by King Soopers after my long run to get a few groceries. Do you ever, because you're ravenously hungry, buy things you didn't know you needed until you see them? I brought home some liverwurst and made a sandwich of it.

https://live.staticflickr.com/65535/52075746751_83a106c928_c.jpg

Liverwurst, pepper jack, red onion, pickle, whole grain mustard, on sourdough bread

Eating this reminds of driving from Logan (Utah) to Salt Lake City with my family to have Sunday dinners with my dad's parents and grandparents. The older Gillieses and Clines almost always had liverwurst and crackers for an appetizer. I don't remember ever disliking it.

by Sean Gillies at May 15, 2022 07:46 PM

May 13, 2022

May 12, 2022

What’s new in a nutshell This is a stability release of the GRASS GIS 8.0 series. The most important changes include the removal of unneeded dynamic loading of GDAL libraries, and a fix in the quantile algorithm. Full list of changes and new features The overview of new features in the 8.0 release series is available at new features in GRASS GIS 8.0. See also our detailed announcement with the full list of changes and bugs fixed at https://github.

May 12, 2022 07:38 PM

May 11, 2022

May 10, 2022

Ci siamo, l’anno 2021 è finito da un po’ e sto scrivendo la lista dei libri che ho letto. Questa lista è dedicata a due persone che mi hanno detto di averla letta negli anni scorsi, A. e M.

Quest’anno ho tenuto traccia dei libri letti e desiderati in un modo nuovo, su un nodo BookWyrm. Ho pensato che avrei potuto leggere 12 libri e ne ho letti 12, ma più che la quantità mi ha dato soddisfazione la continuità. Sono anche riuscito a leggere più lentamente, a rileggere lunghi capitoli e persino un libro intero perché avevo la sensazione di averli letti troppo in fretta.

Avevamo lasciato il nostro lettore alle prese con Seni e uova.

Mieko Kawakami, Seni e uova

Ho impiegato diversi mesi a leggere “Seni e uova”. È un libro monumentale, non solo per le dimensioni.

Cosa vuol dire nascere, cosa vuol dire diventare donna ed esserlo, cosa vuol dire diventare madre, cosa vuol dire diventare padre. Sono tutte domande a cui ho pensato leggendo questo libro.

E poi, anche come si fa a scrivere libri ‒ questo è un altro dei tanti fili che legano le pagine dall’inizio alla fine.

I dialoghi di “Seni e uova” sono più che scambi di parole. Ognuno trova il modo di dire tutto quello che ha da dire, e lo fa in modo straripante, e paralizza essere di fronte a questi lunghi discorsi dove i punti di vista vengono costruiti in modo dettagliato, feroce e liberatorio.

Fatema Mernissi, L’harem e l’occidente

Questa è una lettura che mi era stata consigliata due anni fa nel fediverso. Breve, tagliente e diretto, il resoconto di come Scheherazade si sia annacquata mentre veniva trasportata ad Occidente (Scheherazade goes West, è il titolo originale), come una eroina si sia trasformata in una macchietta di pensierini insulsi nella testa degli uomini bianchi.

Ursula K. Le Guin, Le tombe di Atuan

Come mi sono promesso, ho continuato la lettura della saga, mi sono immerso nel mondo di Terramare e ho trasportato di qua e di là il librone che lo contiene praticamente tutto. Che meraviglia. Ogni libro è una storia dall’inizio alla fine e vive di vita propria, ma i fili che legano luoghi e persone sono lunghi e profondi.

Ursula K. Le Guin, Il signore dei draghi

Questo è il libro della saga che ho trovato più cupo, nonostante Le tombe di Atuan si svolga in gran parte sottoterra. Al centro dell’apoteosi di Ged e della nascita di una nuova guida per Terramare c’è un rapporto di fiducia che è messo in difficoltà e in dubbio dai due protagonisti. E questo è molto più inquietante di una caverna oscura o dell’aldilà.

Ursula K. Le Guin, L’isola del drago

Quando ho iniziato a leggere questo libro ho finalmente capito che Le Guin ha dato una forma di alternanza alle parti della saga, maschile, femminile, maschile, femminile. Questo libro è molto femminile: segue Tenar come protagonista, sia nelle gesta eroiche sia in quelle più semplici nella casa e nella campagna. Tenar salva la piccola Therru, la vittima di una orribile violenza che rimane a lungo traumatizzata, e anche questo è drammaticamente femminile. Finale epico.

Ursula K. Le Guin, I venti di Terramare

L’ultima parte che conclude la saga è di trasfigurazione. Ho letto due volte il libro perché la prima lettura era stata troppo frettolosa, e perché volevo seguire bene il viaggio dei protagonisti. Una storia che inizia con una brocca rotta non può che catturarmi, d’altra parte. Ho fatto un po’ fatica a seguire con attenzione i nomi dei vari Maestri, e ho confuso un po’ il Maestro delle Mani e quello dei Modelli. Sorrido pensando a come viene sconfitta la Morte qui con una vaga reminescenza del finale un po’ ridicolo del Cannocchiale d’ambra (che non rileggerei affatto).

Ursula K. Le Guin, Leggende di Terramare

La raccolta è proprio quello che ci si aspetta per dare spazio alla costruzione del mondo di Terramare. Funziona tutto molto bene, le mie storie preferite qui sono state quelle di Lontra/Medra e di Libellula (che in lingua originale è più direttamente evocativo della protagonista).

James Ellroy, Perché la notte

I personaggi sono piuttosto inquietanti nel loro delirio reciproco, e il modo in cui Lloyd stesso deve ancora una volta seguire istinti fuori dalle regole è particolarmente ben sviluppato.

Mieko Kawakami, Heaven

“Heaven” è appena uscito, il nuovo libro di Mieko Kawakami.

Non è stato facile leggere “Heaven” perché è un romanzo veramente molto diretto e mi ha buttato dentro alcune tematiche che faccio molta fatica ad affrontare serenamente, come il suicidio, la fiducia negli altri, chiudersi in se stessi. Pur essendo molto più breve del precedente romanzo ne ripropone lo stile grandioso, i lunghi mono-dialoghi in cui alcuni personaggi espongono nei minimi dettagli tutto quello che pensano di un determinato argomento, spesso in modo brutale. La mia memoria di alcuni episodi e periodi di bullismo scolastico, ben più leggero di quello affrontato dai protagonisti, è ancora viva. Qui invece andiamo proprio dentro il quotidiano dentro la testa di chi subisce, e per un po’ anche dentro quella di chi massacra gli altri. È molto duro e c’è una parte di violenza quasi grafica. Non conosco abbastanza la cultura né la letteratura giapponese per situare “Heaven” rispetto a questo tema.

Ho voluto finire di leggerlo durante un pomeriggio soleggiato, sul divano, invece che prima di addormentarmi la sera. Ma sono contento di averlo letto.

Paolo Rumiz, Annibale

Che noia. Ripetitivo allo sfinimento, nostalgico di quando si faceva il militare, maschilista.

Gail Honeyman, Eleanor Oliphant sta benissimo

Questo l’ho trovato per caso e mi è piaciuto, un po’ letteratura di genere e un po’ spiazzante e onesto sguardo sulla solitudine, sulla salute mentale.

Elena Ferrante, L’amica geniale

Questo era nella lista dei libri da leggere almeno da due anni. Divorato in modo abbastanza spietato e all’altezza delle aspettative. Per favore non mi parlate di serie TV.

Nel 2022 ho continuato a leggere Elena Ferrante.

by Stefano Costa at May 10, 2022 09:03 PM

Week six was a productive one with four days of running. One tough speed workout, a tempo run in the hills, and one long run along the first ridge on the east edge of the Rocky Mountains. Here are the numbers.

  • 7 hours, 42 minutes running

  • 37.5 miles

  • 4396 ft D+

Saturday I helped at Quad Rock. I transported drop bags from the starting line to the Horsetooth aid station (miles 10 and 40) and then spent 12 hours at the aid station with a stint of watching a tricky trail intersection a mile upstream.

https://live.staticflickr.com/65535/52063049486_89390ff596_c.jpg

Starting line, 5:15 a.m.

https://live.staticflickr.com/65535/52063540570_af9bcbb4c1_c.jpg

Horsetooth aid station, 6:40 a.m.

Mid-day it was warm, about 85° F (30° C). Every runner coming into the aid station a second time looked overheated to some degree, and still had one final climb to go with much of it in full sun. We cooled them down; put ice in water bottles, bandanas, hats; and handed out popsicles. It was super satisfying to see energy and smiles come back to the runners. According to OpenSplitTime, everyone who left our aid station finished. We had one person drop and 4 who didn't make the time cut.

After the trail sweepers came in from the previous station, I helped pack up, and then transported drop bags and the DNF runners to the finish.

Rain fell early Sunday. I waited for it to stop and then went out for an extremely easy 15-mile trail run bracketed by 3-mile bike rides from and to home. I saw lots of kids hiking with their moms.

https://live.staticflickr.com/65535/52063282299_42bcc6b618_b.jpg

Finally a hint of green

by Sean Gillies at May 10, 2022 03:13 AM

May 09, 2022

GeoServer 2.21-RC release is now available with downloads (bin, war, windows), along with docs and extensions.

This is a GeoServer release candidate made in conjunction with GeoTools 27-RC and GeoWebCache 1.21-RC.

  • Release candidates are a community building exercise and are not intended for production use.
  • We ask the community (everyone: individuals, organizations, service providers) to download and thoroughly test this release candidate and report back.
  • Testing priority is the new internationalization support
  • Participating in testing release candidates is a key expectation of our open source social contract. We make an effort to thank each person who tests in our release announcement and project presentations!
  • GeoServer commercial service providers are fully expected to test on behalf of their customers.

Release Candidate Testing Priorities

We would like to ask for your assistance testing the following:

  • Try out the new language chooser, and if you spot any translations you can help out with we would love your assistance translating geoserver.

  • The ability to customize feature types allows for a lot of creativity, please try this out and share your examples

  • This release features a new Log4J logging system. If you only choose between the built-in logging profiles we expect everything to be smooth an uneventful.

    If you have made any custom logging profiles, or customized the built-in logging profiles in place, some additional care is required (see below). We are very interested in this upgrade process and ask for your feedback and testing at this time.

  • For those using GDAL or OGR please check the instructions below on upgrading to GDAL 3.

A reminder that open-source is a community activity and we ask everyone to take part at this time.

Feature Type Customization

We are pleased to share a long-requested feature - the ability to rename attributes and change attribute order when publishing a FeatureType.

Feature type customization

It is also possible to change attribute type, and with the use of ECQL expressions generate new attributes on the fly.

The above example works around the limitations of shapefile to use longer names, and creates a new attribute capital on the fly from an expression, as shown in the following GetFeatureInfo output.

GetFeatureInfo tasmania_cities

This is a great new addition to GeoServer, please see Feture Type Details in the user guide for details.

Thanks to Andrea Aime (GeoSolutions) for proposal GSIP-207 and implementation.

Translations and Language Chooser

A big thanks to Alexandre Gacon and everyone who helped improve GeoServer internationalization for during this release cycle.

To support this activity Andrea Aime has contributed a language chooser to the top of the screen (near the login button).

Language chooser

For more information see Choosing the UI language (User Guide).

  • GEOS-1158 Specify Geoserver UI Language in Configuration

Add Styles support to LayerGroup

Layer groups can now be configured with additional styles, with each style listing a series of layers along with the style used to render each layer.

Layer group styles

This allows a SINGLE or OPAQUE layer group to list alternate styles in addition to the default one. Each alternate style is defined by a named configuration of layers and styles providing a unique visual representation.

In the above example the layer group Tasmania is setup with an alternate “data” presentation, presenting the content with the geoserver default styles point, line and polygon.

Layer group style Tasmania

For more information see Layer Group Styles (User Guide).

  • GEOS-10252 Add Styles support to LayerGroup
  • GEOS-10274 Geofence follow up LayerGroup Style addition

Thanks to Marco Volpini (GeoSolutions) for GSIP-205 proposal and implementation.

GeoPackage WMS and WFS Output

The result of proposal GSIP-206 is the creation of the gs-geopkg-output extension packaging up the WFS and WMS output formats from the geopackage community module.

curl "http://localhost:8080/geoserver/wfs?service=wfs&version=2.0.0&request=GetFeature&typeNames=topp:states&outputFormat=geopkg" -o wfs.gpkg

For more information see Using the GeoPackage Output Extension in the user guide.

  • GEOS-10351 [GSIP 206] Promote GeoPackage WFS and WMS output formats to an extension
  • GEOS-8793 WFS 1.1.0/2.0.0 GeoPackage output wrong Coordinate Order

Thanks to David Blasby and Jody Garnett (GeoCat) for packaging up this work as an extension.

Mark Factory Precedence

When rendering maps with lots of individual graphics, looking up the correct implementation (known as a MarkFactory) can be time consuming.

WMS Settings have new capability to filter out any mark factories not being used, and provide an order to prioritise the ones being used.

MarkFactory precedence

For more information see WMS Web Administration (user guide).

  • GEOS-10230 MarkFactory WMS rendering performance optimization

Thanks to Fernando Mino (GeoSolutions Group) for troubleshooting this performance issue, and proposal GSIP-205 as an optimization.

Log4J 2 Upgrade

The assessment of Log4Shell vulnerability highlighted that although GeoSever was not affected, our use of the older Log4j 1.2 was a notable risk. This discussion resulted a small fundraising effort and proposal to upgrade to Log4j 2.

The result is a small change to the user interface, listing logging profiles by name (previously the file extension was also listed).

Log4j update

Internally this release replaces changes from Log4j 1.2 logging profiles (using properties extension) to Log4j 2 logging profiles (using xml extension):

  • The built-in logging profiles (DEFAULT_LOGGING, PRODUCTION_LOGGING, …) are replaced with new Log4j 2 xml files.

  • Previous custom logging profiles will continue to be available (Log4J 2 has the ability to read the older Log4J 1.2 properties files).

  • If you made any customizations to the built-in profiles, you can recover your changes from backup bak file. You can use this backup as a reference when creating a new xml logging profile, or restore this under a different name which does not conflict with the built-in logging profiles.

    A customization to PRODUCTION_LOGGING.properties will be backed up to PRODUCTION_LOGGING.properties.bak. This can be restored by renaming PRODUCTION_LOGGING.properties.bak to CUSTOM_LOGGING.properties.

In addition to the INFO status messages, you will notice a new CONFIG logging level used during application startup:

CONFIG [org.geoserver] - GeoServer configuration lock is enabled
CONFIG [org.geoserver] - Loading catalog...
...

For more information, and examples of writing on log4J 2 profile, see Logging Settings and Advanced log configuration in the User Guide. Of note is the introduction of a new CONFIG logging level used loading and saving configuration changes.

Thanks to Jody Garnett (GeoCat) for completing this work, and to the following sponsors for supporting this activity.

Logging REST API

For more information please see Logging settings (User Guide) and GeoServer Logging (REST API).

  • GEOS-10368 Logging Controller Addition allows configuration of logging via REST API.

Thanks to Yalın Eren Deliorman for this contribution.

New WPS settings and KML input/output support

A number of improvements have been made to the WPS service:

GDAL 3.x Compatibility

The gdal-output extension is tested against GDAL 3.x series.

Please pay careful attention to the installation instructions, while the extensions includes gdal-3.2.0.jar you should double check the native binaries (included in your Linux distribution or installed by hand) and download an appropriate replacement jar online.

For more information see Installing GDAL native libraries in the User Guide.

  • GEOS-10402 Upgrade imageio-ext to 1.4.0 (tested with gdal 3.2)

Thanks to Andrea Aime (GeoSolutions) for making the ImageIO-EXT release, and Jody Garnett (GeoCat) for GDAL 3.x upgrade and testing.

Improvements and Fixes

New features:

  • GEOS-10228 Add wrap_limit property to wrap the category text values of a legend

Improvements:

  • GEOS-10146 App-schema: support for multiple geometries with different CRS
  • GEOS-10246 jdbcconfig: performance slow-down from unnecessary transactions
  • GEOS-10251 Refactor MapML vocabulary to map- custom elements HTML namespace
  • GEOS-10463 Support WCS default value for Deflate Compression
  • GEOS-10320 Support GetFeatureInfo on raster layers with transformations turning the output into vector
  • GEOS-10405 GetFeatureInfo: Support multiple featureCollections per query layer

Fixes:

  • GEOS-10226 ResourcePool leaves empty files on failure
  • GEOS-10318 CSV output format for complex features doesn’t resolve namespace URIs to prefixes on attributes names
  • GEOS-10235 Prevent double-quote to be specified as CSV separator
  • GEOS-10477 SLD - Validation error on Normalize-node
  • GEOS-10448 GetTimeSeries does not limit number of dates when using a time range request (without period)
  • GEOS-10429 Style validation error using the VendorOption “graphic-margin”
  • GEOS-10318 CSV output format for complex features doesn’t resolve namespace URIs to prefixes on attributes names

Tasks:

  • GEOS-10458 Update jai-ext to 1.1.22
  • GEOS-10446 Upgrade to commons-codec 1.15 version
  • GEOS-10363 Switch from itextpdf to openpdf for PDF map rendering

About GeoServer 2.21

Additional information on GeoServer 2.21 series:

Release notes: ( 2.21-RC )

by Jody Garnett at May 09, 2022 12:00 AM

May 06, 2022

May 05, 2022

The electrons were hardly dry on the JTS Outer and Inner Polygon Hull post when another interesting use case popped up on GIS StackExchange.  The question was how to remove aliasing artifacts (AKA "jaggies") from polygons created by vectorizing raster data, with the condition that the result should contain the original polygon.  

A polygon for Vancouver Island vectorized from a coarse raster dataset.  Aliasing artifacts are obvious.

This problem is often handled by applying a simplification or smoothing process to the "jaggy" polygon boundary.  This works, as long as the process preserves polygonal topology (e.g. such as the JTS TopologyPreservingSimplifier). But generally this output of this process does not contain the input polygon, since the simplification/smoothing can alter the boundary inwards as well as outwards.  

Simplification using TopologyPreservingSimplifier with distance = 0.1.  Artifacts are removed, but the simplified polygon does not fully contain the original.

In contrast, the JTS Polygon Outer Hull algorithm is designed to do exactly what is required: it reduces the number of vertices, while guaranteeing that the input polygon is contained in the result.  It is essentially a simplification method which also preserves polygonal topology (using an area-based approach similar to the Visvalingham-Whyatt algorithm).

Outer Hull using vertex ratio = 0.25.  Artifacts are removed, and the original polygon is contained in the hull polygon.

Here's a real-world example, taken from the GADM dataset for administrative areas of Germany.  The coastline of the state of Mecklenburg-Vorpommern appears to have been derived from a raster, and thus exhibits aliasing artifacts.  Computing the outer hull with a fairly conservative parameter eliminates most of the artifacts, and ensures polygonal topology is preserved.

A portion of the coastline of Mecklenburg-Vorpommern showing aliasing artifacts.  The Outer Hull computed with vertex ratio = 0.25 eliminates most artifacts, and preserves the coastline topology.

Future Work

A potential issue for using Outer Hull as a smoothing technique is the choice of parameter value controlling the amount of change.  The algorithm provides two options: the ratio of reduction in the number of vertices, or the fraction of change in area allowed.  Both of these are scale-independent, and reflect natural goals for controlling simplification.  But neither relate directly to the goal of removing "stairstep" artifacts along the boundary.  This might be better specified via a distance-based parameter. The parameter value could then be determined based on the known artifact size (i.e. the resolution of the underlying grid).  Since the algorithm for Outer Hull is quite flexible, this should be feasible to implement. 

by Dr JTS (noreply@blogger.com) at May 05, 2022 12:15 AM

May 03, 2022

Today, I’m revisiting work from 2017. In Brezina, Graser & Leth (2017), we looked at different ways to determine the width of sidewalks in Vienna based on the city’s street surface database.

Image source: Brezina, Graser & Leth (2017)

Inscribed and circumscribed circles were a natural starting point. Circumscribed or bounding circle tools (the smallest circle to enclose an input polygon) have been commonly available in desktop GIS and spatial databases. Inscribed circle tools (the largest circle that fits into an input polygon) used to be less readily available. Lately, support has improved since ST_MaximumInscribedCircle has been added in PostGIS 3.1.0 (requires GEOS >= 3.9.0).

The tricky thing is that ST_MaximumInscribedCircle does not behave like ST_MinimumBoundingCircle. While the bounding circle function returns the circle geometry, the inscribed circle function returns a record containing information on the circle center and radius. Handling the resulting records involves some not so intuitive SQL.

Here is what I’ve come up with to get both the circle geometries as well as the radius values:

WITH foo AS 
(
	SELECT id, 
		ST_MaximumInscribedCircle(geom) AS inscribed_circle,
		ST_MinimumBoundingRadius(geom) AS bounding_circle
	FROM demo.sidewalks 
)
SELECT
	id,
	(bounding_circle).radius AS bounding_circle_radius,
	ST_MinimumBoundingCircle(geom) AS bounding_circle_geom, 
	(inscribed_circle).radius AS inscribed_circle_radius,
	ST_Buffer((inscribed_circle).center, (inscribed_circle).radius) AS inscribed_circle_geom
FROM foo

And here is how the results look like in QGIS, with purple shapeburst fills for bounding circles and green shapeburst fills for inscribed circles:

References

Brezina, T., Graser, A., & Leth, U. (2017). Geometric methods for estimating representative sidewalk widths applied to Vienna’s streetscape surfaces database. Journal of Geographical Systems, 19(2), 157-174, doi:10.1007/s10109-017-0245-2.

by underdark at May 03, 2022 04:21 PM

May 02, 2022

Our last OSGeo “Code Sprint” event in Bolsena – Italy was in 2019. And this year we are back! The code sprint is planned for an extended period of 10 days, giving all that participate a chance to work and meet a little bit longer. The event in Bolsena is in an Italian monastery (see map location), photos and videos: 2008-video2009-video and 2010-video. The monastery is overlooking Lago Bolsena and offers space to about 25 people in small bedrooms 🙂 . It is one of the coolest places for such an event. Quiet, isolated, simple and serene.

Have a look yourself, including looking at the photo gallery. For what the food concerns, we cook elaborate meals you will definitely enjoy 🙂 . There’s a large dining space, but we will eat outside overlooking the lake if weather permits (very likely in June). You can stay all ten days and food (breakfast, lunch and dinner) is included in the price. As in previous years, we will go out for a city visit and a dinner on one of the evenings.

The event will take place during the whole week, although the emphasis will probably be Monday to Friday for most participants.

More information on the OSGeo wiki.

The post Bolsena Code Sprint 2022 appeared first on GeoCat B.V..

by Michel Gabriel at May 02, 2022 02:23 PM

Dry and windy weather continues here on Colorado's Front Range. I'm not a big fan of running in mud, but we really need some steady, soaking rain here. This dry and desiccating spring is unusual and is challenging our drought-tolerant environment. Many plants are still semi-dormant and the landscape is still mostly a midwinter yellow-brown color and flowers are slow to appear. One exception is the white blossom of the tough-as-nails sand lily (Leucocrinum montanum). This plant is in full bloom locally, as plentiful as ever. I was surprised to learn today that sand lilies are in the same family as asparagus. I saw one clematis blossom in a wetter spot this week, and a few bluebells here and there. In other years, these are erupting everywhere.

Here are the numbers for week five. Season highs all around. I'll go double this distance and 2.5 times the time and climbing one week in July.

  • 8 hours, 35 minutes

  • 43 miles

  • 6010 ft D+

Six days of running, my first substantial speed workout, and two runs with plenty of hills. I'm mostly hiking up the hills and trying to flow down them without braking. In a few weeks I'll start to push it more on the ascents. 5 minute uphill intervals with 1 minute of recovery do amazing things for my fitness.

Saturday, May 7, I will be volunteering all day for the Quad Rock race and will you see you if you pass through the Horsetooth aid station. It's at the 10 and 40 mile marks for the race courses. I'm looking forward to helping you 25 milers settle into the middle of your course and helping you 50 milers get the fuel and what you need for your finish.

by Sean Gillies at May 02, 2022 12:09 AM

April 29, 2022

We are pleased to announce that OTB version 8.0.1 is out ! Ready-to-use binary packages are available on the package page of the website: OTB-8.0.1-Linux64.run (Linux) OTB-8.0.1-Win64.zip (Windows 64 bits) OTB-8.0.1-Darwin64.run (Mac OS) You can also checkout the release branch with git: The documentation for OTB 8.0.1 can be found here. This patch version: Fixes […]

by Julien Osman at April 29, 2022 07:16 AM

April 28, 2022

April 27, 2022

April 25, 2022

The JTS Topology Suite recently gained the ability to compute concave hulls.  The Concave Hull algorithm computes a polygon enclosing a set of points using a parameter to determine the "tightness".  However, for polygonal inputs the computed concave hull does not always respect the polygon boundaries.  So the concave hull may not actually contain the input polygon.

It would be useful to be able to compute the "outer concave hull" of a polygon.  This is a valid polygon formed by a subset of the vertices of the input polygon which fully contains the input polygon.  Vertices can be eliminated as long as the resulting boundary does not self-intersect, and does not cross into the original polygon.

An outer concave hull of a polygon representing Switzerland

As with point-set concave hulls, the vertex reduction is controlled by a numeric parameter. This creates a sequence of hulls of increasingly larger area with smaller vertex counts.  At an extreme value of the parameter, the outer hull is the same as the convex hull of the input.
A sequence of outer hulls of New Zealand's North Island

The outer hull concept extends to handle holes and MultiPolygons.  In all cases the hull boundaries are constructed so that they do not cross each other, thus ensuring the validity of the result.

An outer hull of a MultiPolygon for the coast of Denmark.  The hull polygons do not cross.

It's also possible to construct inner hulls of polygons, where the constructed hull is fully within the original polygon.

An inner hull of Switzerland

Inner hulls also support holes and MultiPolygons.  At an extreme value of the control parameter, holes become convex hulls, and a polygon shell reduces to a triangle (unless blocked by the presence of holes).

An inner hull of a lake with islands.  The island holes become convex hulls, and prevent the outer shell from reducing fully to a triangle

A hull can provide a significant reduction in the vertex size of a polygon for a minimal change in area. This could allow faster evaluation of spatial predicates, by pre-filtering with smaller hulls of polygons.

An outer hull of Brazil provides a 10x reduction in vertex size, with only ~1% change in area.

This has been on the JTS To-Do list for a while (I first proposed it back in 2009).  At that time it was presented as a way of simplifying polygonal geometry. Of course JTS has had the TopologyPreservingSimplifier for many years.  But it doesn't compute a strictly outer hull.  Also, it's based on Douglas-Peucker simplification, which isn't ideal for polygons.  

It seems there's quite a need for this functionality, as shown in these GIS-StackExchange posts (1, 2, 3, 4).  There's even existing implementations on Github: rdp-expansion-only and simplipy (both in Python) - but both of these sound like they have some significant issues. 

Recent JTS R&D (on concave hulls and polygon triangulation) has provided the basis for an effective, performant polygonal concave hull algorithm.  This is now released as the PolygonHull class in JTS.

The PolygonHull API

Polygon hulls have the following characteristics:

  • Hulls can be constructed for Polygons and MultiPolygons, including holes.
  • Hull geometries have the same structure as the input.  There is a one-to-one correspondence for  elements, shells and holes.
  • Hulls are valid polygonal geometries.
  • The hull vertices are a subset of the input vertices.

The PolygonHull algorithm supports computing both outer and inner hulls. 

  • Outer hulls contain the input geometry.  Vertices forming concave corners (convex for holes) are removed.  The maximum outer hull is the convex hull(s) of the input polygon(s), with holes reduced to triangles.
  • Inner hulls are contained within the input geometry.  Vertices forming convex corners (concave for holes) are removed.   The minimum inner hull is a triangle contained in (each) polygon, with holes expanded to their convex hulls.  

The number of vertices removed is controlled by a numeric parameter.  Two different parameters are provided:

  • the Vertex Number Fraction specifies the desired result vertex count as a fraction of the number of input vertices.  The value 1 produces the original geometry.  Smaller values produce simpler hulls.  The value 0 produces the maximum outer or minimum inner hull.
  • the Area Delta Ratio specifies the desired maximum change in the ratio of the result area to the input area.  The value 0 produces the original geometry.  Larger values produce simpler hulls. 
Defining the parameters as ratios means they are independent of the size of the input geometry, and thus easier to specify for a range of inputs.  Both parameters are targets rather than absolutes; the validity constraint means the result hull may not attain the specified value in some cases. 

Algorithm Description

The algorithm removes vertices via "corner clipping".  Corners are triangles formed by three consecutive vertices in a (current) boundary ring of a polygon.  Corners are removed when they meet certain criteria.  For an outer hull, a corner can be removed if it is concave (for shell rings) or convex (for hole rings).  For an inner hull the removable corner orientations are reversed.  

In both variants, corners are removed only if the triangle they form does not contain other vertices of the (current) boundary rings.  This condition prevents self-intersections from occurring within or between rings. This ensures the resulting hull geometry is topologically valid.  Detecting triangle-vertex intersections is made performant by maintaining a spatial index on the vertices in the rings.  This is supported by an index structure called a VertexSequencePackedRtree.  This is a semi-static R-tree built on the list of vertices of each polygon boundary ring.  Vertex lists typically have a high degree of spatial coherency, so the constructed R-tree generally provides good space utilization.  It provides fast bounding-box search, and supports item removal (allowing the index to stay consistent as ring vertices are removed).

Corners that are candidates for removal are kept in a priority queue ordered by area.  Corners are removed in order of smallest area first.  This minimizes the amount of change for a given vertex count, and produces a better quality result.  Removing a corner may create new corners, which are inserted in the priority queue for processing.  Corners in the queue may be invalidated if one of the corner side vertices has previously been removed; invalid corners are discarded. 

This algorithm uses techniques originated for the Ear-Clipping approach used in the JTS PolgyonTriangulator implementation. It also has a similarity to the Visvalingham-Whyatt simplification algorithm.  But as far as I know using this approach for computing outer and inner hulls is novel. (After the fact I found a recent paper about a similar construction called a Shortcut Hull [Bonerath et al 2020], but it uses a different approach).

Further Work

It should be straightforward to use this same approach to implement a Topology-Preserving Simplifier variant using the corner-area-removal approach (as in Visvalingham-Whyatt simplification).  The result would be a simplified, topologically-valid polygonal geometry.  The simplification parameter  limits the number of result vertices directly, or the net change in area.  The resulting shape should be a good approximation of the input, but would not necessarily be either wholly inside or outside.







by Dr JTS (noreply@blogger.com) at April 25, 2022 09:20 PM

I have the day off work today. After saying goodbye to my mother-in-law, who has been visiting, and making lunches for my kids, I drove to Lory State Park for a very nice, hilly trail run (7.75 miles, 1876 ft D+, average grade 4.4% including downhills). I listened to the new King Gizzard & the Lizard Wizard double album, Omnium Gatherum, on the drive and laughed out loud a few times. It's so good. And self-indulgent, which is exactly my mood today.

It's important to refuel after a hard run, so when I got home I made an extra loaded, extra buttery, grilled Cubano-ish sandwich. Did you know the Lucerne cheese company is selling ghost pepper jack? When I saw it in the store I had to grab it. Honestly, it could be hotter.

https://live.staticflickr.com/65535/52030190200_86e72b421a_c.jpg

Black Forest ham, ghost pepper jack, red onion, dill pickle, and mustard on white bread.

by Sean Gillies at April 25, 2022 07:01 PM

Última atualização 2021-12-10 20:18:19.

🏠 Pessoal:

Os últimos meses foram corridos, me descuidei e desde janeiro que não posto nada aqui no blog. Alguns fatos que aconteceram nesse tempo:

  • Cortei os cabelos depois de 2 anos, mas já estou sentido falta dos cabelos longos.
  • Estou procurando uma casa na vizinhança. Em breve novidades…

⚒ Profissional:

Houve mudanças no ambiente de trabalho, mudei de setor e agora tenho mais responsabilidades. Hoje estou numa posição de gestão, o que me afastou da parte técnica do setor de GEO.

❤ Comunidade:

Estou participando do comitê geral do FOSS4G 2022, que irá acontecer em Florença-Itália, mais informações no site oficial do evento.

📚 Leitura:

Li o livro ONDE MORREM OS AVIÕES, do Lito Sousa do canal: aviões e música.

Meu perfil no Skoob

Nota de transparência: os links dos livros acima têm código de afiliado. Clicando neles, os preços não mudam, mas posso ganhar uma comissão da Amazon.

📺 Assistindo:

Do que assistir destaco o documentário Downfall: The Case Against Boeing interessante documentário sobre os incidentes envolvendo o Boeing 737 MAX.

De filmes assisti The Highwaymen, filme da Netflix sobre a caçada ao casal de ladrões Bonnie e Clyde. Ainda sobre no tema policial indico a série The Great Robbery of Brazil’s Central Bank que narra os fatos do assalto ao banco central de Fortaleza. Outro filme que assisti foi Gemini Man e não recomendo.

No momento estou acompanhando Better Call Saul: Season 6 , para mim a melhor série do momento. Também estou assistindo Moon Knight .

🎧 Ouvindo:

Não consigo indicar outra coisa a não ser o podcast Mano a Mano, original Spotify com Mano Brown. De fato um programa que me fez ampliar a visão e o debate trazendo diversidade de ideias e pensamentos com profundidade e respeito. 

Frase do momento


“Minha alucinação é suportar o dia a dia, e o meu delírio é a experiência com coisas reais.”

— Belchior

The post Abril / 2022 appeared first on Narcélio de Sá.

by Narcélio de Sá at April 25, 2022 01:45 AM

April 24, 2022

In week three I did six consecutive days of running. Track intervals on Wednesday. Singletrack tempo on Thursday. Back-to-back 2+ hour runs Saturday and Sunday. The weather was cool and windy and I fought with a sinus infection for motivation. Happily, 30 minutes of running never failed to make me feel better.

  • 7 hours, 58 minutes

  • 40.5 miles

  • 5262 ft D+

Half of my week four training was in San Francisco. Tuesday I went along the Embarcadero and up the Filbert Steps to the Coit Tower on Telegraph Hill. Wednesday I went to Christmas Tree Point on the north summit (922 ft) of the Twin Peaks via Corona Heights.

  • 5 hours, 8 minutes

  • 28.3 miles

  • 2854 ft D+

https://live.staticflickr.com/65535/52027834514_ef7a7011d6_b.jpg

Downtown San Francisco from Corona Heights

The little peak of Corona Heights is made out of chert, a dense and glassy rock. A quarry has uncovered an unusually large section of rock polished by fault motion. Geologists call this a slickenside. I saw a coyote at the base of the hill. I've seen signs warning about them around the city before, but this was the first time I've seen a coyote in San Francisco. Tom MacWright gave Corona Heights a 5 for its views. The view of the city and bay is great, but there's no view of the ocean, so I might give it a 4.8.

I concur with Tom's 6/5 score for running at Twin Peaks. The access from the north, at least, is still car free. There's a longish 8% grade that is super satisfying to finish. I try to make it up there every time I visit San Francisco.

by Sean Gillies at April 24, 2022 11:01 PM

April 23, 2022

Since last week’s post, I’ve learned that there is an official OGC Moving Features JSON Encodings repository with more recent sample datasets, including MovingPoint, MovingPolygon, and Trajectory JSON examples.

The MovingPoint example seems to describe a storm, including its path (temporalGeometry), pressure, wind strength, and class values (temporalProperties):

You can give the current implementation a spin using this MyBinder notebook

An exciting future step would be to experiment with extending MovingPandas to support the MovingPolygon MF-JSON examples. MovingPolygons can change their size and orientation as they move. I’m not yet sure, however, if the number of polygon nodes can change between time steps and how this would be reflected by the prism concept presented in the draft specification:

Image source: https://ksookim.github.io/mf-json/

by underdark at April 23, 2022 07:43 PM

April 21, 2022

This is a guest post by Mickael HOARAU @Oneil974

As an update of the tutorial from previous years, I created a tutorial showing how to make a simple and dynamic color map with charts in QGIS.

In this tutorial you can see some of interesting features of QGIS and its community plugins. Here you’ll see variables, expressions, filters, QuickOSM and DataPlotly plugins and much more. You just need to use QGIS 3.24 Tisler version.

Here is the tutorial.

by oneil974 at April 21, 2022 06:17 PM

Since I got the basic sanity checking PHP script running without crashing, there was one problem remaining around MgByteReader contents being read incorrectly. This turned out to be another out-of-date SWIG typemap around byte arrays. Once that got fixed we are able to read out the contents of the MgByteReader without content corruption. This is an important thing to test because a lot of results from the MapGuide API (XML results, rendered map images, map plots, etc) come in the form of MgByteReaders. If we can't read content out of these readers without data corruption we have major problems, but that is fortunately not the case (so far!).

With that out of the way, it was onto to the current PHP test suite for the MapGuide API and getting it fixed up and passing.

But before we go there, I wanted to check if our new PHP binding was leaking memory in the most obvious cases. To that effect, I built our PHP binding with reference counting diagnostics enabled and wrote this PHP script.

  1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
<?php

function exceptionTest() {
/*
Expected release book-keeping:
MgException - 1
*/
try {
$rid = new MgResourceIdentifier("iamnotvalid");
} catch (MgException $ex) {

}
}

function throwingFunction() {
$agfRw = new MgAgfReaderWriter();
$wktRw = new MgWktReaderWriter();

$rid = new MgResourceIdentifier("iamnotvalid");
}

function exceptionTest2() {
/*
Expected release book-keeping:
MgAgfReaderWriter - 1
MgWktReaderWriter - 1
MgException - 1
*/
try {
throwingFunction();
} catch (MgException $ex) {

}
}

function exceptionTest3() {
/*
Expected release book-keeping:
MgAgfReaderWriter - 1
MgWktReaderWriter - 1
MgException - 2
*/
try {
$rid = new MgResourceIdentifier("iamnotvalid");
} catch (MgException $ex) {

}

try {
throwingFunction();
} catch (MgException $ex) {
//Previous iterations of the binding would leak the previous $ex when
//reusing the variable in this catch block
//If the release book-keeping for MgException is 2, then this is no
//longer the case
}
}

function geometryXformTest() {
/*
Expected release book-keeping:
MgCoordinateSystemFactory - 1
MgCoordinateSystem - 2
MgTransform - 1
MgWktReaderWriter - 1
MgPoint - 2
MgCoordinateXY - 2
*/
$csFactory = new MgCoordinateSystemFactory();
$cs1 = $csFactory->CreateFromCode("LL84");
$cs2 = $csFactory->CreateFromCode("WGS84.PseudoMercator");
$xform = $csFactory->GetTransform($cs1, $cs2);
$wktRw = new MgWktReaderWriter();
$pt = $wktRw->Read("POINT (1 2)");
$coord = $pt->GetCoordinate();
echo "Point (".$coord->GetX().", ".$coord->GetY().")\n";
$pt = $pt->Transform($xform);
$coord = $pt->GetCoordinate();
echo "XPoint (".$coord->GetX().", ".$coord->GetY().")\n";
}

function connectionServiceTest() {
/*
Expected release book-keeping:
MgUserInformation - 1
MgSiteConnection - 1
MgProxyResourceService - 1
MgProxyFeatureService - 1
MgProxyRenderingService - 1
MgProxyMappingService - 1
MgProxyKmlService - 1
MgProxyDrawingService - 1
MgProxyTileService - 1
MgProxyProfilingService - 1
*/

$userInfo = new MgUserInformation("Anonymous", "");
$site = new MgSiteConnection();
$site->Open($userInfo);

$service1 = $site->CreateService(MgServiceType::ResourceService);
$service2 = $site->CreateService(MgServiceType::FeatureService);
$service3 = $site->CreateService(MgServiceType::RenderingService);
$service4 = $site->CreateService(MgServiceType::MappingService);
$service5 = $site->CreateService(MgServiceType::KmlService);
$service6 = $site->CreateService(MgServiceType::DrawingService);
$service7 = $site->CreateService(MgServiceType::TileService);
$service8 = $site->CreateService(MgServiceType::ProfilingService);
}

function functionWithParam($userInfo) {
$userInfo->SetLocale("en");
}

function parameterPassingTest() {
// We want to check parameters of Mg* objects passed do not encounter
// refcounting shenanigans, but if they do, we want to make sure that
// proper increment/decrement is happening so that the MgUserInformation
// is released to 0 refcount (and thus deleted) when this function returns

/*
Expected release book-keeping:
MgUserInformation - 1
*/
$userInfo = new MgUserInformation();
functionWithParam($userInfo);
echo "Locale: " . $userInfo->GetLocale() . "\n";
}

MgInitializeWebTier("C:\\mg-4.0-install\\Web\\www\\webconfig_dev.ini");

echo "=========== BEGIN - exceptionTest() ===========\n";
exceptionTest();
echo "=========== END - exceptionTest() ===========\n";

echo "=========== BEGIN - exceptionTest2() ===========\n";
exceptionTest2();
echo "=========== END - exceptionTest2() ===========\n";

echo "=========== BEGIN - exceptionTest3() ===========\n";
exceptionTest3();
echo "=========== END - exceptionTest3() ===========\n";

echo "=========== BEGIN - connectionServiceTest() ===========\n";
connectionServiceTest();
echo "=========== END - connectionServiceTest() ===========\n";

echo "=========== BEGIN - parameterPassingTest() ===========\n";
parameterPassingTest();
echo "=========== END - parameterPassingTest() ===========\n";

echo "=========== BEGIN - geometryXformTest() ===========\n";
geometryXformTest();
echo "=========== END - geometryXformTest() ===========\n";

?>

The basic tenets behind this test script are:
  • If we new any Mg* class, we expect it to be released (reference count dropped by 1) once the respective variable goes out of scope. We consider the script to not be leaking if every "new" statement has a corresponding release as reported by our refcounting diagnostics.
  • Any non-primitive value returned by any class in the MapGuide API is also released once its respective captured variable goes out of scope
  • If an MgException is thrown and we catch it, it is released once we exit the respective catch block
  • If we have multiple try/catch blocks on MgException and we use the same variable name for the caught exception, we're not leaking on subsequent catch blocks (this used to happen when we were generating the PHP bindings on the older SWIG/PHP version)
  • Finally any MapGuide API object passed as parameters to other functions does not get touched by reference counting. But if it does, we expect that the reference count for that object is the same before and after the function call.
Running this script produces the following output



The key things to notice in the script output is that every variable assignment is being properly released upon exiting the function and anything that we new'd explicitly was released with a 1 -> 0 reference count transition. When an object's reference count drops to 0, it will be de-allocated on the C++ side. The script output is proof that we aren't leaking objects we explicitly new'd up from the PHP script, they are being properly de-allocated. Some objects (like the MgCoordinateSystem) release and do not drop to 0 reference count, but that is okay because I know that the MapGuide API caches MgCoordinateSystem instances for performances reasons, so they aren't leaking instances. They can live beyond the scope of the function call where I initially requested for such objects.

So now knowing that we aren't leaking for the general cases, we can move onto the main PHP test suite.

Running the main PHP test suite it for the first time (in a long time) showed some PHP errors around ambiguous overloaded methods. All of these errors are because PHP 7/8 must've tightened up some of the type checking and passing null as a parameter to a method that has many overloaded signatures may cause confusion. Either that or our heavily modified version of SWIG that we previously used to generate the PHP binding had custom codegen to specifically handle overloaded method resolution.

Regardless, the fix was simple enough. Where passing null would cause ambiguity, we just needed to add the necessary cast (in our case, it was casting to string) to make sure the correct method signature is invoked. Once that fix was applied, the test suite ran to completion ...


... with a metric crapton of test failures :(

Now to figure out why these test cases are failing, which is where we come to the subject of this post.

Anytime in the past, when I need to debug a PHP script, it was a case of sprinkling var_dump and print_r statements everywhere, running it and try to comprehend the now extra verbose output, rinsing and repeating until I found the problem. Basically, printf debugging.

I had heard about a thing called xdebug, but I had no idea how to pair it with an IDE (and what IDE to even use?) to get the nice integrated debugging experience I take for granted with .net, Java or C++. Debugging PHP script felt so primitive because I simply didn't know better and getting proper debugging set up just sounded too hard.

Now, times have changed. We have Visual Studio Code. It has a nice ecosystem of extensions for working with PHP that supports the version of PHP that we are now targeting. If debugging C++ code inside docker containers turned out to be such a piece of cake, then surely the same can be said for using VSCode to debug PHP scripts with breakpoints, variable inspection and the whole works!

So I searched for a VSCode extension that could help me debug PHP, and the first result was this PHP debug adapter. I followed its instructions to the letter.

I went to the xdebug install wizard and pasted by dumped phpinfo details


Once submitted, the wizard figured out what version of PHP I had and gave me the download link for the xdebug extension to download and further instructions on what php.ini changes I needed to make


Then back to the VSCode extension instructions, I also updated php.ini to activate extra xdebug settings

1
2
xdebug.mode = debug
xdebug.start_with_request = yes

Then in VSCode, I opened the PHP script I intend to run and debug, and created a new launch.json file for my active workspace.


The newly installed VSCode PHP debug extension must've kicked in at this point because it created a launch.json with 3 xdebug launch presets ready to go. I modify the "Launch currently open script" preset to add in the extra command-line arguments needed by my PHP script.


Once I saved the launch.json, I open the PHP script again, stuck some breakpoints, switched to the debug tab, making sure to use the "Launch currently open script" preset.


I press the play button to start debugging and lo and behold! We're now breaking and stepping through PHP code and inspecting variables!


Now I can focus on figuring out why there are so many test failures, at a much faster pace than what I would've normally done in the past!

No doubt when MapGuide Open Source 4.0 is finally out the door, I'll have to write up a variation of this post on how to set up VSCode with PHP debugging for your PHP-based MapGuide applications. Because nobody should be sprinking var_dumps and print_r statements to debug PHP code like I have done in the past when this is many orders of magnitude faster and more productive!

by Jackie Ng (noreply@blogger.com) at April 21, 2022 04:21 PM

April 20, 2022

The Dream

Over a year ago Chris Holmes, the driving force behind the “cloud optimized GeoTIFF” (COG) and “spatio-temporal asset catalog” (STAC) standards that are sweeping the “cloud optimized raster” data management world, asked me what I thought the story of a similar “cloud optimized vector” format might look like.

And I thought about COG, which is really just a very very old format (GeoTIFF) with its bytes rearranged so that the order of bytes in the file matches the likely order in which they will be accessed (blocks of bands, and within the bands, squares of pixels), and I thought I had the perfect, maximally snarky answer:

Dude, shape file is already a cloud-native format.

Now, this might seem counter-intuitive, but hear me out:

  • Shape format hails from the early 90’s, when hard-disks spun very slowly, and the limiting factor for data access was usually I/O. Which is much like “cloud optimized” range access over HTTP: seeks are expensive, but block reads are cheap.
  • Shape format already divies up the attributes and shapes into separate files, so you can render one without reading the other.
  • Shape format is already “network safe”, with endianness defined in the format.
  • Shape format is already universally supported. The specification is 24 years old, and it has been the de facto interchange format for so long that people make a joke out of it.

In short, shapefile already looks a lot like GeoTIFF, the founding format of the “cloud optimized geospatial” movement.

COSHP

Let’s Get Real

So, what is missing to make “cloud optimized shapefile” a reality?

Well, in order to spatially search a shapefile you need a spatial index. There is no index format in the formal Esri specification, and the Esri sbx index format is proprietary (though largely reverse engineered at this point) but the open source world has had a shape file index format for 20 years: the “QIX file”.

You generate a QIX file using the shptree utility in Mapserver. You can also generate one in GDAL. You can also get GeoTools to generate one.

With a QIX file, the “shape file” now consists of four files:

  • shp, the binary shapes
  • dbf, the table of attributes
  • shx, a file index of the byte offsets of each shape in the shp file
  • qix, a spatial index of the shapes

The next trick, just as in COG, is to put the main data files (shp, dbf and shx) into the same order they are likely to be searched in: spatial order.

Since we already have a spatial index (qix), we can get the files in spatial order by re-writing them in the same order they appear in the index.

Initially I told Chris that this could be done with the Mapserver sortshp utility, however I was mistaken: sortshp sorts the file in attribute order.

To make “cloud optimized shape file” a reality, first we need a new utility program that sorts the shp, shx and dbf files into qix order.

We need: coshp!

coshp is just a re-working of the sortshp utility, but instead of sorting the output file by attribute it sorts it using the index tree as the driver. This results in shp and dbf files where shapes that are “near” in space are also “near” in the byte-stream of the file. This will reduce the number of random reads necessary to access portions of the file using a bounding box search.

Not Even Remotely Done

One of the quiet secrets of the “cloud optimized” geospatial world is that, while all the attention is placed on the formats, the actual really really hard part is writing the clients that can efficiently make use of the carefully organized bytes.

For example, the fancy demonstrations of “pure client-side” COG browsers require a complete GeoTIFF reader in Javascript, along with some extra “cloud” smarts to know what pieces of data to cache and what to treat as transient info during rendering.

So, spatially sorting a shape file is a necessary, but not at all sufficient condition to create an actual “cloud optimized shapefile”, because for it to be practically useful, there needs to be at a minimum a client-side javascript reader.

That means javascript that can:

To be truly useful, the javascript should probably include enough cloud smarts to read and opportunistically cache pieces of the files: headers at a minimum, but also reading in branches of the qix and shx indexes on-demand.

To make things marginally easier, I have “documented” the QIX format. It’s an ugly beast, but it is possible to traverse it without deserializing the whole thing.

It’s a challenging problem, but I hope there is someone with enough nostalgia for old formats and thirst for glory to make it happen.

April 20, 2022 08:00 AM

April 19, 2022

Recently I have gotten back into the grind of MapGuide development pushing to clear the final hurdle of having functional PHP bindings for the MapGuide API.

Since I last touched this code, several things have changed.

  • We originally wanted to target PHP 7.x, but this is no longer the latest. PHP 8.1.x is the latest. This release of PHP 8.1 for Windows is also built with the same MSVC compiler we use for MapGuide and FDO (MSVC 2019), which means the stars have aligned again where we can source official windows PHP binaries for bundling.
  • Our last attempt used SWIG 4.0.2 (the most recent stable release). This release unfortunately cannot generate functional bindings for PHP 8, so our hand has been forced and we are now using and assuming current git master of SWIG for generating PHP bindings. The latest git master also no longer generates a [PHP extension .dll + .php proxy script] combination and goes back to generating just a PHP extension .dll, which is great because that how our (now) legacy PHP binding was generated.
Given these environmental changes, we've re-activated the PHP binding work with a semi-clean slate. A lot of hacks and workarounds that were in place because the previous SWIG generated a [PHP extension .dll + .php proxy script] combination have been removed.

The PHP binding work has now advanced to the stage where we can run this basic sanity test script against a binary PHP 8.1.4 distribution on Windows with no errors or crashes.

  1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
<?php

echo "Initializing web tier";
try {
MgInitializeWebTier("C:\\mg-4.0-install\\Web\\www\\webconfig_dev.ini");
} catch (MgException $initEx) {
echo "Init failure!";
die;
} catch (Exception $ex) {
echo "[php]: Exception: " . $ex->getMessage() . "\n";
die;
}
echo "[php]: Initialized\n";

// We haven't and shouldn't need to require/include constants.php
// they are now baked into the PHP extension along with the other
// MapGuide API proxy classes
echo "[php]: Testing some constants\n";
echo " - " . MgMimeType::Agf . "\n";
echo " - " . MgMimeType::Kml . "\n";
echo " - " . MgMimeType::Mvt . "\n";
echo " - " . MgImageFormats::Png . "\n";
echo " - " . MgImageFormats::Gif . "\n";
echo " - " . MgLogFileType::Authentication . "\n";
echo " - " . MgServerInformationProperties::ClientOperationsQueueCount . "\n";

$user = new MgUserInformation("Anonymous", "");
// Basic set/get
$user->SetLocale("en");
echo "[php]: Locale is: " . $user->GetLocale() . "\n";
$conn = new MgSiteConnection();
$conn->Open($user);
// Create a session repository
$site = $conn->GetSite();
$sessionID = $site->CreateSession();
echo "[php]: Created session: $sessionID\n";
$user->SetMgSessionId($sessionID);
// Get an instance of the required services.
$resourceService = $conn->CreateService(MgServiceType::ResourceService);
echo "[php]: Created Resource Service\n";
$mappingService = $conn->CreateService(MgServiceType::MappingService);
echo "[php]: Created Mapping Service\n";
$resId = new MgResourceIdentifier("Library://");
echo "[php]: Enumeratin'\n";
$resources = $resourceService->EnumerateResources($resId, -1, "");
echo $resources->ToString() . "\n";
echo "[php]: Coordinate System\n";
$csFactory = new MgCoordinateSystemFactory();
echo "[php]: CS Catalog\n";
$catalog = $csFactory->GetCatalog();
echo "[php]: Category Dictionary\n";
$catDict = $catalog->GetCategoryDictionary();
echo "[php]: CS Dictionary\n";
$csDict = $catalog->GetCoordinateSystemDictionary();
echo "[php]: Datum Dictionary\n";
$datumDict = $catalog->GetDatumDictionary();
echo "[php]: Coordinate System - LL84\n";
$cs1 = $csFactory->CreateFromCode("LL84");
echo "[php]: Coordinate System - WGS84.PseudoMercator\n";
$cs2 = $csFactory->CreateFromCode("WGS84.PseudoMercator");
echo "[php]: Make xform\n";
$xform = $csFactory->GetTransform($cs1, $cs2);
echo "[php]: WKT Reader\n";
$wktRw = new MgWktReaderWriter();
echo "[php]: WKT Point\n";
$pt = $wktRw->Read("POINT (1 2)");
$coord = $pt->GetCoordinate();
echo "[php]: X: ".$coord->GetX().", Y: ".$coord->GetY()."\n";
$site->DestroySession($sessionID);
echo "[php]: Destroyed session $sessionID\n";
echo "[php]: Test byte reader\n";
$bs = new MgByteSource("abcd1234", 8);
$content = "";
$br = $bs->GetReader();
$buf = "";
while ($br->Read($buf, 2) > 0) {
echo "Buffer: '$buf'\n";
$content .= $buf;
}
echo "[php]: Test byte reader2\n";
$bs2 = new MgByteSource("abcd1234", 8);
$content = "";
$br2 = $bs2->GetReader();
$buf = "";
while ($br2->Read($buf, 3) > 0) {
echo "Buffer: '$buf'\n";
$content .= $buf;
}
$agfRw = new MgAgfReaderWriter();
echo "[php]: Trigger an exception\n";
try {
$agfRw->Read(NULL);
} catch (MgException $ex) {
echo "[php]: MgException caught\n";
echo "[php]: MgException - Code: ".$ex->GetExceptionCode()."\n";
echo "[php]: MgException - Message: ".$ex->GetExceptionMessage()."\n";
echo "[php]: MgException - Details: ".$ex->GetDetails()."\n";
echo "[php]: MgException - Stack: ".$ex->GetStackTrace()."\n";
echo "Is standard PHP exception too? " . ($ex instanceof Exception) . "\n";
} catch (Exception $ex) {
echo "[php]: Exception: " . $ex->getMessage() . "\n";
}
echo "[php]: Trigger another exception\n";
try {
$r2 = new MgResourceIdentifier("");
} catch (MgException $ex2) {
echo "[php]: MgException caught\n";
echo "[php]: MgException - Code: ".$ex2->GetExceptionCode()."\n";
echo "[php]: MgException - Message: ".$ex2->GetExceptionMessage()."\n";
echo "[php]: MgException - Details: ".$ex2->GetDetails()."\n";
echo "[php]: MgException - Stack: ".$ex2->GetStackTrace()."\n";
echo "Is standard PHP exception too? " . ($ex2 instanceof Exception) . "\n";
} catch (Exception $ex) {
echo "[php]: Exception: " . $ex->getMessage() . "\n";
}

echo "Press any key to exit\n";
readline();

?>

This script is a "sanity check" in that it is testing the key aspects of the MapGuide API that *must* function properly.
  • We can init the web tier
  • We can create a site connection
  • We can request instances of MgService-derived subclasses with proper downcasting (eg. if we ask for a resource service, we get a MgResourceService and not its parent MgService)
  • We can catch MgExceptions and inspect all relevant properties from it.
  • We can access constants. Notice we aren't require/include-ing constants.php? That's because we've found a way to get SWIG to bake all MapGuide constants into the PHP extension itself so this file is no longer necessary!
  • We can interact with MgByteReader contents properly
Currently the sanity check script is failing on the last point and I strongly suspect it is another case of SWIG typemaps that were relevant for PHP7 but no longer valid for PHP8. We also need to make sure that this script doesn't leak memory. Also I've yet to get this binding building on Linux nor have I ran this sanity check script on Linux.

Once this is taken care of, the long journey begins to make sure everything else is working in order of importance:
  1. Our PHP test suite passes
  2. Our PHP web tier applications (site administrator, schema report, AJAX viewer) are functional
  3. Fusion is functional
  4. We can build updated Apache and PHP end-to-end on Linux
  5. All of the above is functional in both IIS and Apache web server configurations on both Windows and Linux
Then, and only then we can start thinking about a new MapGuide Open Source 4.0 preview/beta release.

by Jackie Ng (noreply@blogger.com) at April 19, 2022 04:30 PM