Friday, March 24, 2017

Runnable vs. Thread in Java

Runnable (an interface you implement) vs. Thread (an class you extend): Basically, always use Runnable. It's less tied to a concurrency model (meaning it works with Threads and Futures) and allows reuse of threads. Executors work with Threads and Futures... well, technically only with Runnables, but Thread and FutureTask both implement Runnable. A big fan of futures suggested 'Future f = new FutureTask<object>(runnable, null)', but I think that's pretty ugly.

Thursday, March 23, 2017

ACID / BASE /CAP

ACID


The claim to fame for relational databases is they make the ACID promise:


Atomicity - A transaction is all or nothing
Consistency - Only valid data is written to the database
Isolation - Pretend all transactions are happening serially and the data is correct
Durability - What you write is what you get. Data won't disappear.


The "problem" with ACID is that it may be giving you too much at the cost of performance. Fulfilling these promises has a big impact to scalability. It trips you up when you are trying to scale a system across multiple nodes.


Down time is generally unacceptable for cloud applications. So your system needs to be reliable. Reliability requires multiple nodes to handle machine failures. To make scalable systems that can handle lots and lots of reads and writes you need many more nodes.


Once you try to scale ACID across many machines you hit problems with network failures and delays. The algorithms don't work in a distributed environment at any acceptable speed.

BASE

The types of large systems based on CAP aren't ACID. They are BASE (har har):

Basically Available - System seems to work all the time
Soft State - It doesn't have to be consistent all the time
Eventually Consistent - Becomes consistent at some later time


CAP


If you can't have all of the ACID guarantees, it turns out you can have two of the following three characteristics:


Consistency - Your data is correct all the time. What you write is what you read.
Availability - You can read and write your data all the time
Partition Tolerance - If one or more nodes fails, the system still works and becomes consistent when the system comes on-line.

Wednesday, March 22, 2017

Type Erasure in Java

When generics are used, they're converted into compile-time checks and execution-time casts. Type erasure means that at execution time, there's no way of figuring out that is a String because that information has been erased. Type Erasure exists in order to keep java bytecode backwards compatible with old JVM versions. Instead of implementing generics through erasure, generics could be implemented through “reification.” These would be called reified generics and would retain the type information. For a pretty good introductory exploration of generics in Java, see http://beust.com/weblog/2011/07/29/erasure-vs-reification/.

Tuesday, March 21, 2017

Lat / Lon and Bounding Areas

To add to the post made yesterday, I thought I'd add some notes and code for some latitude / longitude bounding area logic.

Deciding if a point is within a bounding box on a globe

You need two cases to account for the special case where the bounding box crosses over the 180 degree meridian.  Here's some sample code:

  1. if( upperLeftLong > lowerRightLong &&  
  2.     latitude <= upperLeftLat &&  
  3.     latitude >= lowerRightLat &&  
  4.     longitude >= upperLeftLong &&  
  5.     longitude <= lowerRightLong )  
  6. {  
  7.   //Special case when the 180 degree meridian crossed  
  8.   //The point is within the bounding area  
  9. }  
  10. else if ( latitude <= upperLeftLat &&  
  11.           latitude >= lowerRightLat &&  
  12.           longitude >= upperLeftLong &&  
  13.           longitude <= lowerRightLong )  
  14. {  
  15.   //Normal case  
  16.   //The point is within the bounding area  
  17. }  
  18. else  
  19. {  
  20.   //The point is not within the bounding area  
  21. }  
Deciding if a point is within a bounding circle on a globe:

In the project I was working on, someone else had written a function that determines whether or not a coordinate is within a bounding circle on Earth.  The bounding circle was defined by a center coordinate and a radius.  The function was not always working and the original author was long gone, so I looked at it myself.  It turns out the problem was that the equation being used was for a 2 dimensional plane, which the Earth is not.  So I just needed to replace it with an equation for determining if a point is within a circle on a sphere. This is what I found:

acos(sin(lat1)*sin(lat2)+cos(lat1)*cos(lat2)*cos(lon2-lon1))*EarthRadius <= Radius

The latitudes and longitudes must be in radians. This is the spherical law of cosines and should work well for our purposes. The Haversine Formula seems to be more precise, but more computationally expensive, and we're not using small values anyway. For further optimization, see: http://www.movable-type.co.uk/scripts/latlong-db.html

Monday, March 20, 2017

Lat/Lon + Distance + Heading --> Lat/Lon

TLDR: Java's % operator is not really a mod operation, so if you're doing the mod operations with potentially negative numbers, you might want to be careful.

I was working on a mapping related application. I came across an unexpected difficulty that I thought would be worthwhile to share.

One of my functions was supposed to return the latitude and longitude of a location a specific distance and heading from a provided initial point. I found a very good resource for equations dealing with latitudes and longitude (http://williams.best.vwh.net/avform.htm), which was very helpful for this application in general. I found the equation I wanted, made sure it made sense to me, then coded it up in Java.

However, there were lots of problems during testing with negative numbers. It turns out the problem was that the mod operator in Java is not a true mod operation, so my calculations dealing with negative lat / longs were wrong. These following two blog posts do a good job explaining the problem, so I'm just going to copy and paste them here:


From http://www.velocityreviews.com/forums/t388345-mod-of-a-negative-number.html:
Strictly speaking, there is no "mod" operator in Java. % is defined to
be "remainder." It is the same as modulo for positive operands.

For integer operands, Java's % is designed to maintain the identity:
(a/b)*b+(a%b) is equal to a.

Java integer division rounds towards zero, so -5/-2 is 2.

( (-5)/(-2) ) * (-2) + (-1) is -5.

See the JLS for more information,
http://java.sun.com/docs/books/jls/s...oc.html#239829

Patricia Shanahan
The JLS example was also illuminating:

5 % 3 produces 2
5 % (-3) produces 2
(-5) % 3 produces -2
(-5) % (-3) produces -2

This got me thinking...

Suppose a and b are integers.

We have two special cases:

1.If b == 0, then a % b is NaN (JLS specification).

2.If nonzero b divides a, then a % b == 0 (In particular, every nonzero b divides 0, so 0 % b == 0).  Which leaves us with nonzero b does not divide a.

Then,

a % b > 0 if a > 0

and

a % b < 0 if a < 0.

Hence, if nonzero b does not divide a, then the sign of a % b equals the sign of a.

You can see this formally by noting:

(a / b) * b + a % b == a

implies

a % b == a - (a / b) * b

and if nonzero b does not divide a, then

Math.abs((a / b) * b) < Math.abs(a),

due integer division rounding towards zero.

kstahmer   
I think those two posts should make the problem very clear.  Please note that this issue with the mod operator is present in many other programming languages as well. The reason it was done this way is because it's faster to compute through hardware. Anyway, the equation for the mod operation we wanted to use instead of Java's % operator is "mod(y,x) = y - x*floor(y/x)".

That's about it from me today.  In case you need it, here's my final function for calculating a coordinate a given distance and heading away from an initial coordinate.  I hope it works right now, I'll leave it to you to import the proper packages:

  1. /** 
  2.  * Gets the latitude and longtitude of the coordinate a given distance from 
  3.  * a given initial coordinate on a given heading. 
  4.  * 
  5.  * Formula followed from: http://williams.best.vwh.net/avform.htm#LL 
  6.  * On that page, it is listed as "Lat/lon given radial and distance" 
  7.  * A main difference to note is that this method uses km instead of miles 
  8.  * and uses nautical miles for the distance. 
  9.  * 
  10.  * @param latitude The latitude of the intial point. 
  11.  * @param longitude The longitude of the intial point. 
  12.  * @param distance The given distance in km. 
  13.  * @param heading The given heading in degrees. 
  14.  * @return Point 
  15.  **/  
  16. static Point getCoordinateFromInitialPoint(double latitude,  
  17.      double longitude, double distance, double heading)  
  18. {  
  19.   double d = distance/1000;  
  20.   double lat1 = Math.toRadians(latitude);  
  21.   double lon1 = Math.toRadians(longitude);  
  22.   double lat2 = 0.0;  
  23.   double lon2 = 0.0;  
  24.   double crs = Math.toRadians(heading);  
  25.   
  26.   lat2=Math.asin(Math.sin(lat1)*Math.cos(d/RADIUS)+Math.cos(lat1)*  
  27.         Math.sin(d/RADIUS)*Math.cos(crs));  
  28.   
  29.   lon2=Math.atan2(  
  30.         Math.sin(crs)*Math.sin(d/RADIUS)*Math.cos(lat1),  
  31.         Math.cos(d/RADIUS)-Math.sin(lat1)*Math.sin(lat2));  
  32.   
  33.   // normalize lon2 to -180...+180  
  34.   // Do not use % operator for mod here because it does not behave as  
  35.   // required for this function when the inputs are negative numbers, as  
  36.   // explained in http://williams.best.vwh.net/avform.htm#Math and  
  37.   // http://en.wikipedia.org/wiki/Modulo_operation  
  38.   double x = lon1-lon2+Math.PI;  
  39.   double y = 2*Math.PI;  
  40.   double modxy = x-y*Math.floor(x/y);  
  41.   lon2 = modxy - Math.PI;  
  42.   
  43.   lat2 = Math.toDegrees(lat2);  
  44.   lon2 = Math.toDegrees(lon2);  
  45.   
  46.   return new Point(lat2, lon2);  
  47. }  

Friday, March 17, 2017

My Notes on the Java RMI and Static Ports

[I wrote this 9/23/2008 with Java 1.6, but it's probably still accurate]

Java Remote Method Invocation (RMI) is a technology that makes it very easy to implement server client architectures. Java RMI lets you to make method calls to remote objects without having the client treat the remote object very differently from a local object. I've been using RMI a fair amount recently, so I thought I'd go over some of the more interesting things I've learned. Primarily, how it works and how to control the ports it uses.

A Brief Overview about how RMI Works

First there are the "Stubs."  Stubs are essentially specialized proxies to a specific server. Internally, they hold the IP and Port to a server. They know how to make requests to the server and how to process the response from that server. They're Java objects and are used exactly the same as a local object once you've obtained the stub.

Then there's the "RMI Registry."  The RMI Registry is a server that stores name to stub matchings. All its clients know the IP and Port number for the Registry. The clients contact the registry to get stub objects for servers whose IP and Port number they don't necessarily know. The Registry listens on port 1099 unless configured otherwise.

Finally, there's a server run for each stub that's registered in the RMI Registry.

And so the following happens when RMI is used:
  1. The RMI Registry that stores name to stub pairings is created.
  2. Whenever a new name and stub is bound in the Registry, a matching server is created (usually on a dynamic port) for the stub to communicate with.
  3. The client knows the IP and Port of an RMI Registry. The client uses that IP and Port to ask the registry for the stub matching a certain name.
  4. When a client calls a method on the stub, the stub contacts the matching server to process the request. The matching server processes the response and returns a result to the stub. The stub in turn processes the response and returns a value as if the method was called locally.
This really is a brief overview, for a lot more detail about the specifics, give http://www.developer.com/java/ent/article.php/10933_3455311_1 a thorough read. One thing not mentioned is that RMI is multi-threaded. I'm not entirely sure about the following point, but I think that a new thread is automatically created for every stub request. I don't know of anyway to make RMI non multi-threaded (in case you can't have every client sending information to the same JVM.)

But You Don't Care How It Works, do you?

The great thing about RMI is that the programmer doesn't really have to worry about the network communications at all. They just need to get a Java object (the stub) from the Registry and make method calls on it. The stub, skeleton, and the registry do all the network stuff for them. So all a programmer really cares about is implementing RMI. For that information, Java's own tutorial is probably the best source. So go and read it at http://java.sun.com/docs/books/tutorial/rmi/server.html. Really, the rest of this article won't make sense without knowing how to use RMI.

Stop Randomly Choosing Ports!!!

For the project I was working on, it was important to control the IP and the Ports RMI used. I was using Solaris 10 with Trusted Extensions and wanted Cross Domain capabilities, which required me to explicitly identify certain ports as "Multi Level Ports." You can see how dynamically assigning ports to RMI servers was no good for me. Furthermore, I was using several different IP Addresses on the machine and did not want RMI to to accept connections on all of them. So what this really came down to was controlling the sockets that are normally dynamically created by the Registry and RMI Servers.

I wrote my own RMIServerSocketFactory to gain control over the sockets in RMI. It is as follows:
CustomSocketFactory.java

view plaincopy to clipboardprint?
  1. ///////////////////////////////////////////////////////////////////////////  
  2. //  
  3. // CustomSocketFactory - This class is used to provide client and server  
  4. //   sockets with configurable ports and ip addresses.  By default,  
  5. //   RMISocketFactory chooses these dynamically.  Note that the backlog  
  6. //   for ServerSockets is hardcoded.  
  7. //  
  8. ///////////////////////////////////////////////////////////////////////////  
  9.   
  10. import java.io.IOException;  
  11. import java.io.Serializable;  
  12. import java.net.InetAddress;  
  13. import java.net.ServerSocket;  
  14. import java.net.Socket;  
  15. import java.rmi.server.RMISocketFactory;  
  16.   
  17. /** 
  18.  * CustomSocketFactory 
  19.  */  
  20. public class CustomSocketFactory extends RMISocketFactory implements Serializable  
  21. {  
  22.     /** 
  23.      * The InetAddress to be used for the Sockets this factory serves. 
  24.      */  
  25.     private InetAddress ipInterface = null;  
  26.   
  27.     /** 
  28.      * The port to be used with Sockets this factory serves.  By 
  29.      * default this value is set to -1 (which is an impossible port), 
  30.      * which tells the factory to allow the port be be dynamically determined. 
  31.      */  
  32.     private int port = -1;  
  33.   
  34.     /** 
  35.      * Default Constructor.  IP and Port will be dynamically assigned. 
  36.      */  
  37.     public CustomSocketFactory() {}  
  38.   
  39.     /** 
  40.      * Constructor that specifies the INetAddress to use.  The port will still 
  41.      * be served dynamically. 
  42.      * 
  43.      * @param ipInterface The InetAddress to be associated with Sockets served by 
  44.      * this factory. 
  45.      */  
  46.     public CustomSocketFactory(InetAddress ipInterface)  
  47.     {  
  48.         this.ipInterface = ipInterface;  
  49.     }  
  50.   
  51.     /** 
  52.      * Constructor that specifies the INetAddres and port to use. 
  53.      * 
  54.      * @param ipInterface The INetAddress to be associated with Sockets served by 
  55.      * this factory. 
  56.      * @param port The port to be associated with Sockets served by this 
  57.      * factory. 
  58.      */  
  59.     public CustomSocketFactory(InetAddress ipInterface, int port)  
  60.     {  
  61.         this.ipInterface = ipInterface;  
  62.         this.port = port;  
  63.     }  
  64.   
  65.     /** 
  66.      * Returns a ServerSocket with the ip address and port used dependant 
  67.      * on the constructor used. 
  68.      * 
  69.      * @param port The dynamic port to be used for the socket served.  This 
  70.      * is not used if the constructor specifying a port was used 
  71.      * for this factory. 
  72.      * @return A socket. 
  73.      */  
  74.     public ServerSocket createServerSocket(int port) throws IOException  
  75.     {  
  76.         ServerSocket serverSocket = null;  
  77.   
  78.         //Decides whether or not to use the supplied port  
  79.         ifthis.port != -1 )  
  80.         {  
  81.             port = this.port;  
  82.         }  
  83.   
  84.         try  
  85.         {  
  86.             //NOTE: The backlog is hardcoded to 50 here.  This was  
  87.             //  done for no particular reason and could be dynamically  
  88.             //  set by making new method calls.  
  89.             serverSocket = new ServerSocket(port, 50, ipInterface);  
  90.         } catch( IOException e ) {  
  91.             throw e;  
  92.         }  
  93.   
  94.         return serverSocket;  
  95.     }  
  96.   
  97.     /** 
  98.      * Returns a Socket with the ip address and port used dependant 
  99.      * on the constructor used. 
  100.      * 
  101.      * @param port The dynamic port to be used for the socket served.  This 
  102.      * is not used if the constructor specifying a port was used 
  103.      * for this factory. 
  104.      * @param dummy This INetAddress is not used. 
  105.      * @throws Exception Throws for any exception. 
  106.      * @return A socket. 
  107.      */  
  108.     public Socket createSocket( String dummy, int port) throws IOException  
  109.     {  
  110.         ifthis.port != -1 )  
  111.         {  
  112.             port = this.port;  
  113.         }  
  114.   
  115.         return new Socket( ipInterface, port );  
  116.     }  
  117.   
  118.     /** 
  119.      * Returns true if the object passed in is identical to this object. 
  120.      * 
  121.      * @param that The object to be compared with this object. 
  122.      * @return True if equal, otherwise false. 
  123.      */  
  124.     public boolean equals( Object that )  
  125.     {  
  126.         return (that != null && that.getClass() == this.getClass() );  
  127.     }  
  128. }  
Now I simply specified that SocketFactory when I created the RMI Registry and the stubs/servers! Here's a very simple example:

AddServerInterface.java
view plaincopy to clipboardprint?
  1. import java.rmi.Remote;  
  2. import java.rmi.RemoteException;  
  3.   
  4. public interface AddServerInterface extends Remote  
  5. {  
  6.     public int add( int a, int b ) throws RemoteException;  
  7. }  
  1. import java.net.InetAddress;  
  2. import java.rmi.RemoteException;  
  3. import java.rmi.registry.LocateRegistry;  
  4. import java.rmi.registry.Registry;  
  5. import java.rmi.server.UnicastRemoteObject;  
  6.   
  7. public class AddServer implements AddServerInterface  
  8. {  
  9.     public AddServer() throws Exception  
  10.     {        
  11.         //Set up the RMI.  Use hard coded values for this example.  
  12.         String registryIp = "10.10.1.1";  
  13.         int registryPort = 3645;  
  14.         String serverIp = "10.10.1.1";  
  15.         int serverPort = 3646;  
  16.           
  17.         //Create the Add Server, supplying the Socket Factories to assign our IP and Port.  
  18.         CustomSocketFactory clientSocketFactory = new CustomSocketFactory( InetAddress.getByName(serverIp), serverPort );          
  19.         CustomSocketFactory serverSocketFactory = new CustomSocketFactory( InetAddress.getByName(serverIp), serverPort );  
  20.         AddServerInterface stub = (AddServerInterface)UnicastRemoteObject.exportObject(this,serverPort,clientSocketFactory,serverSocketFactory);  
  21.           
  22.         //Get or Create the RMI registry.  
  23.         Registry registry = null;          
  24.         try  
  25.         {  
  26.             //Create Registry  
  27.             CustomSocketFactory sf = new CustomSocketFactory( InetAddress.getByName(registryIp), registryPort );  
  28.             registry = LocateRegistry.createRegistry( registryPort, null, sf );  
  29.         }  
  30.         catch( RemoteException re )  
  31.         {  
  32.             //Registry was already created, so just connect  
  33.             registry = LocateRegistry.getRegistry(registryPort);    
  34.         }  
  35.           
  36.         //Bind the Add Server to the registry  
  37.         registry.rebind( "AddServer", stub );   
  38.     }  
  39.       
  40.     public int add( int a, int b ) throws RemoteException  
  41.     {  
  42.         return a+b;  
  43.     }  
  44.       
  45.     public static void main( String args[] ) throws Exception  
  46.     {  
  47.         try  
  48.         {              
  49.             AddServer addServer = new AddServer();  
  50.         } catch ( Exception e ) {  
  51.             System.out.println( "Main error: " );  
  52.             e.printStackTrace();  
  53.         }  
  54.     }  
  55. }  
So now the registry listens on 10.10.1.1/3645 and the Add Server listens on 10.10.1.1/3646. It works like a charm. Go ahead, try an example client:

  1. //Required for RMI Support  
  2. import java.net.InetAddress;  
  3. import java.rmi.Remote;  
  4. import java.rmi.registry.LocateRegistry;  
  5. import java.rmi.registry.Registry;  
  6.   
  7. public class AddClient  
  8. {  
  9.     public AddClient() {}  
  10.       
  11.     public static void Main( String args[] )  
  12.     {  
  13.         try   
  14.         {          
  15.            //Connect to the add server.    
  16.            Registry registry = LocateRegistry.getRegistry( "10.10.1.1"3645 );  
  17.            Remote remote = registry.lookup( "AddServer" );  
  18.            if( remote == null )  
  19.            {  
  20.               System.out.println( "Could not find it" );  
  21.            }  
  22.            AddServerInterface addServer = (AddServerInterface)remote;  
  23.            
  24.        System.out.println( "1 + 1 = " + addServer.add( 11 ) );             
  25.         } catch ( Exception e ) {  
  26.            e.printStackTrace();  
  27.         }          
  28.     }  
  29. }  

Thursday, March 16, 2017

Compress multiple files with 7zip command line

Just a quick shell command to use 7zip to individually compress multiple files using the same password. Wow, that was a mouthful.

From windows shell:
for f in [files]; do 7z a -tzip -mem=AES256 -mx=9 -p[password] "/[destination directory]/`basename $f`.zip" "$f"; done;

Example:
for f in /export/home/me/*.txt; do 7z a -tzip -mem=AES256 -mx=9 -pPa55w0rd! "/export/home/me/zips/`basename $f`.zip" "$f"; done;

So, in this example, a new password protected zip file will be created for every text file in /export/home/me. The new zip file will have the same name as before, just with a .zip extension instead of a .txt extension. So "/export/home/me/a.txt" becomes "/export/home/me/zips/a.zip". The password will be "Pa55w0rd!". Just remove the obvious options if you don't want your archive to be password protected.