Skip to main content

Changes in Java to support supplementary Unicode characters

Support for supplementary characters might need changes in the Java language as well as the API. A few questions come to mind.

  • How do we support supplementary characters at the primitive level (char is only 16 bits)?
  • How do we support supplementary characters in low level API's (such as the static methods of the Character class) ?
  • How do we support supplementary characters in high level API's that deal with character sequences?
  • How do we support supplementary characters in Java literals?
  • How do we support supplementary characters in Java source files?

The expert commitee that worked on JSR-204 dealt with all these questions and many more (I'm sure) . After deliberating as well as experimenting with how the changes would affect code, they came up with the following solution.

The primitive char was left unchanged. It is still 16 bits and no other type has been added to the Java language to support the supplementary range of unicode characters.

 Low level API's, such as static methods of the Character class, accepted the char primitive type before support for supplementary characters was provided in Java. However, since Java 5.0, methods such as isLetter(...) of the Character class provide an overloaded method that accepts an int representing the code point, along with the earlier method that accepted a char.

 
JavaCharacterAPI.JPG 

 

High level API's will continue to work "as is" for most developers. They represent character sequences as UTF-16 sequences. Some methods in String and StringBuffer now have parrallel methods to work with code points. Some such methods are codePointAt(...) , codePointBefore(...), and codePointCount(). For example the codePointCount() method returns the number of code points in a String, which may not be the same as the number of characters in the String, if some characters are from the supplementary range and are represented as surrogate pairs.

 

JavaStringMethodsForUnicode.JPG 

 

Identifiers in Java can contain any letter or digit. Many supplementary characters are letters or digits. To allow supplementary characters to be used in identifiers, the Java compiler and other tools were modified to use different API methods (isJavaIdentifierPart(int), isJavaIdentifierStart(int)).

Since we need to support supplementary characters all the way, they also need to be supported in Java source files. I will discuss how to include unicode characters in Java source files and get them to compile using the Java compilers -encode option, in the next blog post.

While I was reading about encoding, I came accross this interesting blog post that describes a situation when an I18N enables Java program ceased to work after the build machine was moved from a Windows box to a Red Hat box. The reason of course was encoding related issues.

 



Note: This text was originally posted on my earlier blog at http://www.adaptivelearningonline.net
Labels:

Comments

Post a Comment

Popular posts from this blog

Running your own one person company

Recently there was a post on PuneTech on mom's re-entering the IT work force after a break. Two of the biggest concerns mentioned were : Coping with vast advances (changes) in the IT landscape Balancing work and family responsibilities Since I have been running a one person company for a good amount of time, I suggested that as an option. In this post I will discuss various aspects of running a one person company. Advantages: You have full control of your time. You can choose to spend as much or as little time as you would like. There is also a good chance that you will be able to decide when you want to spend that time. You get to work on something that you enjoy doing. Tremendous work satisfaction. You have the option of working from home. Disadvantages: It can take a little while for the work to get set, so you may not be able to see revenues for some time. It takes a huge amount of discipline to work without a boss, and without deadlines. You will not get the benefits (insuranc...
10 comments
Read more

Testing Groovy domain classes

If you are trying to test Grails domain class constraints by putting your unit test cases in the 'test/unit' directory, then your tests will fail because the domain objects will not have the 'valdate' method. This can be resolved in two ways: Place the test cases inside test/integration (which will slow things down) Use the method 'mockForConstraintsTests(Trail)' to create mock method in your domain class and continue writing your test cases in 'test/unit' What follows is some example code around this finding. I am working on a Groovy on Grails project for a website to help programmers keep up and refresh their skills. I started with some domain classes and then moved on to write some unit tests. When we create a Grails project using grails create-app , it creates several directories, one of which is a directory called 'test' for holding unit tests. This directory contains two directories, 'unit', and 'integration' for unit and ...
11 comments
Read more

Moving from 2006 to 2007

In 2006 I did a lot research on elearning, instructional technology, and psychology of learning. After much deliberation I came to the conclusion that learning is all about taking in new information, reflecting upon it, practicing it, conversing with others and then sharing. This entire process not only strengthens the learners own understanding of a topic, but also helps the community by knowledge sharing. Learning can be very effective when done as a conversation, rather than a monologue (with either a teacher or a book). Which is why some experts suggest that you should talk to your book while learning, by making notes. Having a conversation helps the learners derive knowledge rather than just having it spoonfed to them. New media technology gives us an entire new way to have these conversations, by blogging and podcasting. I have written here , and here about the benefits of blogging. In the spirit of enabling learning through conversations, I created a website - Adaptive Learni...
5 comments
Read more