Creating a terrain mask shader

In this blog post, we’ll talk about creating a mask shader using ShaderGraph and applying it on our Mapbox Unity SDK terrain. As you might know, we talked about using Mapbox Unity SDK in URP projects so now that we got that out of the way, we’ll be able to use ShaderGraph for this. (ShaderGraph doesn’t support standard render pipeline and only works with URP/HDRP).

I used a similar effect in my Scifi map project to create a hexagon shaped map and a lot of people asked me about it since then.

Scifi map project and map mask shader — You can find a Twitter “Moment” on this project here

As you can guess, this is a common case when you want to put a map on another object, like table or an AR surface. And there are many different ways to do this but in this post, we’ll got for a shader based mask using mask texture and ShaderGraph.

Wait, what is a mask?

According to google; “A texture mask is a grayscale texture used to limit the effects of underlying element”. Or in other words, it’s a texture which dictates where underlying thing will be visible or not. Details of it depends on the implementation I guess but they are all more or less same, be it photoshop mask, 3d shader, UI mask etc etc.

how masks and mask shaders works — Luminosity of the mask texture pixel will be used as opacity of corresponding base texture pixel.

In our case, base “effect” will be a mesh with satellite imagery from Mapbox Unity SDK, mask will be a black and white hexagon jpg and “+” part will be a shader where we tell GPU to draw satellite imagery pixels or not.

Let’s get started!

First two elements, base map and mask should be fairly easy. For this post we can use ZoomableMap demo scene inside Mapbox Unity SDK. It’s a top down map we can pan and zoom so it’s exactly what we need.
For the mask, we will use a very basic hexagon image as seen above. You can find the image here.

But for sure, shader is where all the magic will be. And now we’ll dive into that. I’ll go over the basics fast.
First of all, we decided to use URP and ShaderGraph for this already so let’s be sure those two packages are loaded and project settings are set to use URP properly.

Then let’s create a new shader asset by choosing Create > Shader > PBR Graph from the context menu in project window. And a new material to use that shader. Shader is all empty at the moment but doesn’t matter, we’ll get to that in a minute. Finally, select the Map object in the scene, find the Tile Material field under Abstract Map script, General > Other settings and set your new material there. As the name suggests, this will tell Mapbox Unity SDK to use our new material for terrain meshes.

creating mask shader files — Map settings should look like this now.

And if you run the scene now, you should get a white/gray view. That’s expected though as our shader is all empty and it’s just rendering the default color on the mesh.

Let there be colors!

So let’s start with something simple. Let’s create a new Texture field; you can name it anything you want but this is very important; the Reference has to be _BaseMap. Mapbox Unity SDK will be using this name to pass textures to GPU (see Using Mapbox Unity SDK with URP post for details) so if it doesn’t match in code and shader, it won’t work.

Then sample that texture and use it for Albedo channel. I also decreased Smoothness to zero as I just hate that (so just a visual preference).

mask shader graph color channel — Now you should have imagery applied on mesh.

Color is working now so let’s start with trying the same for alpha.
Create a new Texture field; Reference isn’t that critical for this yet but we’llneed to remember it later. Sample that and use it for Alpha as seen in image below. Our material Surface is set to Opaque by default, let’s change that to Transparent.
Finally, find the material in your project and set the Mask texture to the black and white hexagon image.

mask shader view — Another crazy screenshot where I’m trying to show everything at once…

Hmm, we have multiple hexagons on our map now. But why?

When we sample mask texture like that, and I mean using default UV values for sample node by “that”, we’re actually doing a sampling in UV space. Remember maps contains bunch of square tiles and each tile has a separate mesh, texture and of course UV space going from (0,0) to (1,1).

masking and uv spaces — On the left you can see the UV coordinates of individual tiles
On the right is what happens when we apply mask using those coordinates.

We just want one hexagon in a one specific place though, one specific place in world space. So we should somehow map current pixel’s position (relative to mask position) to UV coordinate. Or maybe if we think vice versa, there’ll be a UV space/square (0-1) where we want to place the mask and we need to convert current pixel’s position into that space.

Relative vertex positions — The one hex mask we want. And we need to check position of all pixels in reference to that.
Only shown 6 in the image but of course shader will work for all pixels on terrain mesh.
Reds will be invisible, blues will be visible,

It’s actually not too difficult; first we subtract mask center position from pixel world space position to find the relative position (while lines in image above).
Then we divide that vector by the mask scale, because that rectangle can be any size, right? If it was big enough, it would have contained those red points as well. This will give us a UV space coordinate.
Finally, we’ll rotate this UV vector as your table/environment/surface might not always align with your mask texture. Imagine having a rectangular mask for a rectangular table but it’s rotated in scene.

I posted the whole calculation above but I’ll go through it one by one as well. After all this time using graph editors, I’m still not sure if I love them or not. But I must say, they feel extremely readable (up until some certain point). Easy to get the flow/idea in general.

First part is the positioning. We subtract mask position from pixel position and divide it by the mask scale. Any value inside mask area should be in [-0.5, 0.5] range now.

We convert this position to vector2 and drop Y axis. We’re working in Z plane so Y axis is useless in this case.
Then we remap from [-0.5,0.5] to [0,1] to move it into UV range. Using remap here is a little overkill and this can be optimized but I preferred to do it this way for simplicity.

We rotate it around [0.5, 0.5] because that’s the center of the UV space & mask texture.
Also converting from ccw to cw and degree to radian before that.

And finally we’re saturating (clamp(0,1)) the value. This will enable/disable texture repeating. But it’ll only work if texture asset is set to Repeat mode as well. Otherwise it won’t repeat no matter what this option is.

Now it should technically work but there’s one more small problem. We set our parameters mask position, mask scale etc. in material settings but when we run the project Mapbox SDK clones that material and creates instances of it for each tile because they all have to have different color maps (satellite imagery). Now they all have different materials and if you ever want to tweak some numbers and style it in runtime, you will have to select them all one by one and change the values separately.

Terrain tiles with different mask parameters — Tiles with different scale values. Kinda hard to style like this.

So it should all work now and this is totally optional, but I really would like a script where I can tweak all numbers.

Global Shader Variables

We’ll use global shader variables for this. They are static global variables of the shader which you can set from anywhere and they are shared between all instances of the entity. So that’s exactly what we want for fields like mask position, scale etc.

First thing we need to do is to disable inspector support for these parameters. Inspector overrides the values set by code so if they are exposed in the inspector, our code won’t work at all.
You can find this on shader graph blackboard. For every parameters, there should be an Exposed checkbox. If you uncheck that, that parameter will not be on the inspector.

Then we need a C# script to get&set values;

public class ShaderController : MonoBehaviour
    {
        public Texture2D MaskTexture;
        public float MaskScale;
        public float MaskRotation;
        public bool RepeatMask;

        private void Update()
        {
            SetValues();
        }

        private void OnValidate()
        {
            SetValues();
        }

        public void SetValues()
        {
            Shader.SetGlobalTexture("_MaskTexture", MaskTexture);
            Shader.SetGlobalFloat("_MaskScale", MaskScale);
            Shader.SetGlobalFloat("_MaskRotation", MaskRotation);
            Shader.SetGlobalVector("_ReferencePosition", transform.position);
            Shader.SetGlobalInt("_RepeatMask", RepeatMask ? 1 : 0);
        }

        private void OnDrawGizmos()
        {
            var v1 = transform.position + Quaternion.Euler(0, MaskRotation, 0) * new Vector3(- MaskScale/2, 0,  - MaskScale/2);
            var v2 = transform.position + Quaternion.Euler(0, MaskRotation, 0) * new Vector3(- MaskScale/2, 0,  + MaskScale/2);
            var v3 = transform.position + Quaternion.Euler(0, MaskRotation, 0) * new Vector3(+ MaskScale/2, 0,  + MaskScale/2);
            var v4 = transform.position + Quaternion.Euler(0, MaskRotation, 0) * new Vector3(+ MaskScale/2, 0,  - MaskScale/2);

            Gizmos.DrawLine(v1, v2);
            Gizmos.DrawLine(v2, v3);
            Gizmos.DrawLine(v3, v4);
            Gizmos.DrawLine(v4, v1);
        }
    }

It should be straight-forward. OnValidate would have been enough in most cases but this scripts uses objects’ own position for mask position and OnValidate doesn’t fire on position changes so I added Update call as well.

As I said before, this is totally optional but if you like to play around with values as much as I do, you might need it.