These houses are mostly made out of glass so that you can see the lava or water, and also let in light.
For underwater houses, you can use sponges to make space for you to place the blocks without drowning or having to keep going up for air. You could also place the blocks and keep going up for air until you have made a shell. From there, you fill your house up with dirt or any other block of your choice so that there is no water in your house.
For an underlava house, you can build your shelter anyway you want it, and then place lava above your structure using lava buckets.