Skip to main content

Changes in Java to support supplementary Unicode characters

Support for supplementary characters might need changes in the Java language as well as the API. A few questions come to mind.

  • How do we support supplementary characters at the primitive level (char is only 16 bits)?
  • How do we support supplementary characters in low level API's (such as the static methods of the Character class) ?
  • How do we support supplementary characters in high level API's that deal with character sequences?
  • How do we support supplementary characters in Java literals?
  • How do we support supplementary characters in Java source files?

The expert commitee that worked on JSR-204 dealt with all these questions and many more (I'm sure) . After deliberating as well as experimenting with how the changes would affect code, they came up with the following solution.

The primitive char was left unchanged. It is still 16 bits and no other type has been added to the Java language to support the supplementary range of unicode characters.

 Low level API's, such as static methods of the Character class, accepted the char primitive type before support for supplementary characters was provided in Java. However, since Java 5.0, methods such as isLetter(...) of the Character class provide an overloaded method that accepts an int representing the code point, along with the earlier method that accepted a char.

 
JavaCharacterAPI.JPG 

 

High level API's will continue to work "as is" for most developers. They represent character sequences as UTF-16 sequences. Some methods in String and StringBuffer now have parrallel methods to work with code points. Some such methods are codePointAt(...) , codePointBefore(...), and codePointCount(). For example the codePointCount() method returns the number of code points in a String, which may not be the same as the number of characters in the String, if some characters are from the supplementary range and are represented as surrogate pairs.

 

JavaStringMethodsForUnicode.JPG 

 

Identifiers in Java can contain any letter or digit. Many supplementary characters are letters or digits. To allow supplementary characters to be used in identifiers, the Java compiler and other tools were modified to use different API methods (isJavaIdentifierPart(int), isJavaIdentifierStart(int)).

Since we need to support supplementary characters all the way, they also need to be supported in Java source files. I will discuss how to include unicode characters in Java source files and get them to compile using the Java compilers -encode option, in the next blog post.

While I was reading about encoding, I came accross this interesting blog post that describes a situation when an I18N enables Java program ceased to work after the build machine was moved from a Windows box to a Red Hat box. The reason of course was encoding related issues.

 



Note: This text was originally posted on my earlier blog at http://www.adaptivelearningonline.net

Comments

Popular posts from this blog

Running your own one person company

Recently there was a post on PuneTech on mom's re-entering the IT work force after a break. Two of the biggest concerns mentioned were : Coping with vast advances (changes) in the IT landscape Balancing work and family responsibilities Since I have been running a one person company for a good amount of time, I suggested that as an option. In this post I will discuss various aspects of running a one person company. Advantages: You have full control of your time. You can choose to spend as much or as little time as you would like. There is also a good chance that you will be able to decide when you want to spend that time. You get to work on something that you enjoy doing. Tremendous work satisfaction. You have the option of working from home. Disadvantages: It can take a little while for the work to get set, so you may not be able to see revenues for some time. It takes a huge amount of discipline to work without a boss, and without deadlines. You will not get the benefits (insuran

Testing Groovy domain classes

If you are trying to test Grails domain class constraints by putting your unit test cases in the 'test/unit' directory, then your tests will fail because the domain objects will not have the 'valdate' method. This can be resolved in two ways: Place the test cases inside test/integration (which will slow things down) Use the method 'mockForConstraintsTests(Trail)' to create mock method in your domain class and continue writing your test cases in 'test/unit' What follows is some example code around this finding. I am working on a Groovy on Grails project for a website to help programmers keep up and refresh their skills. I started with some domain classes and then moved on to write some unit tests. When we create a Grails project using grails create-app , it creates several directories, one of which is a directory called 'test' for holding unit tests. This directory contains two directories, 'unit', and 'integration' for uni

Some thoughts on redesigning education

Some time back I read a blog post on redesigning education. It asked some very good questions. Stuff which I had been thinking of myself. I left my thoughts on the blog, but I would also like to start a conversation around these ideas with those who read this blog as well. I would like to know what other people think of the issue of redesigning (college) education. I have often thought about how college education can be improved. To answer this question, we first have to ask a very basic question. What is the purpose of education? To me, we need education for 3 things: To learn more about the world around us To lead positive constructive lives To earn a good living / fulfill our ambitions I think education has to a large extent evolved to fulfill #3 (with a bias towards earning a comfortable living). The semester system, along with multiple choice tests, and grading, has made our education system into an assembly line. Students are pushed into the assembly line, given classes, admini