[jcifs] Character Encioding Problems on OS/390
Allen, Michael B (RSCH)
Michael_B_Allen at ml.com
Fri Oct 25 08:01:48 EST 2002
> -----Original Message-----
> From: Bigham, Ted [SMTP:TBigham at kmart.com]
> Sent: Thursday, October 24, 2002 4:29 PM
> To: 'jcifs at samba.org'
> Subject: RE: [jcifs] Character Encioding Problems on OS/390
>
> Thanks for the input Allen.
>
> I think you might be looking at things the same way I used to before I
> started working with Java in the EBCDIC world. The big thing I learned is
> that Java doesn't internally keep the format for strings that are created.
> It always converts them to Unicode. You probably already knew that, but the
> implication it causes is that toAscii( ) or toEbcdic( ) methods would have
> to return byte arrays not strings, since ascii and ebcdic only relate to
> bytes not string.
>
You're right. I see.
> The strings that go into and out of jCIFS are actually perfectly valid
> Unicode strings. The UniAddress.getByName() call actually works and finds
> my domain controller. Everything is good up until jCIFS does a getBytes()
> instead of a getBytes("UTF-8"), to create the SMB packet. I had hoped to
> cause a different flow of logic and used a domain name instead (that's where
> the true came from, I never took It back out) but it threw a host not found
> exception. My problem isn't getting the right strings into jCIFS, it's
> getting the strings and byte arrays it internally creates to add the "UTF-8"
> encoding. I was digging around in the source and it looks like changing it
> everywhere wouldn't be all that big of a chore. I just wish someone else
> already did it.
>
Or you mean use char[] for all array work and then at the last minute create a
String from it and do getBytes( "ISO-8859-1" ). I still don't understand were the
UTF-8 comes in though. Also you sound like you know of all of the locations in
the code were these changes would need to occur but almost all operations
are byte oriented. Can you give me a few example locations?
>
> -----Original Message-----
> From: Allen, Michael B (RSCH) [mailto:Michael_B_Allen at ml.com]
> Sent: Wednesday, October 23, 2002 7:40 PM
> To: 'Bigham, Ted'; 'jcifs at samba.org'
> Subject: RE: [jcifs] Character Encioding Problems on OS/390
>
>
>
> > -----Original Message-----
> > From: Bigham, Ted [SMTP:TBigham at kmart.com]
> > Sent: Wednesday, October 23, 2002 12:32 PM
> > To: 'jcifs at samba.org'
> > Subject: [jcifs] Character Encioding Problems on OS/390
> >
> > (using version 7.0b5)
> > I'm trying to authenticate my client connections against a domain
> controller
> > from a servlet. The server (servlet engine) runs on and OS/390 machine
> with
> > the file.encoding property set to Cp1047 which is IBM EBCDIC. This
> setting
> > is required to read the .jsp files in their native format, and changing
> > isn't (and shouldn't have to be) be an option.
> >
> Unfortunately there are many places where the jCIFS code assumes
> it's using an ASCII
> compatible character encoding. This is largely because CIFS is
> inherently an ASCII
> compatible and UCS-2LE (also ASCII compatible) protocol. This is
> illustrated clearly in
> the jcifs.util.Log code and it's printHexDump method which is not
> printing some of the
> logging messages correctly. I cannot fix this. It would require very
> pervasive changes. It
> is very unusual to run the VM in the EBCDIC locale and yet do IO
> with a non-EBCDIC
> peers like the CIFS protocol. For webpages this isn't so bad because
> you can just run
> everything through a converter since HTTP is entirely text based.
> You can try to convert
> *all* strings going into and coming out of JCIFS. In theory that
> *should* work. You could
> write a little toEbcdic and toAscii methods easily and do:
>
> public static void main(String[] args) throws Exception
> {
> jcifs.util.Log.setMask(jcifs.util.Log.ALL);
> UniAddress domainController =
> UniAddress.getByName(toAscii("148.162.36.216"));
> byte[] challenge =
> SmbSession.getChallenge(domainController);
>
> System.out.println(toEbcdic(domainController));
> for (int i=0; i < challenge.length; i++)
> System.out.print(challenge[i] + ":");
> System.out.println();
> }
>
> But this will probably require some judicious hacking and in the end
> there's a good chance
> it just wouldn't work because of some assumption made in the jCIFS
> code. Also note, you
> do not need the second 'true' parameter for getByName. You're not
> looking up a workgroup
> or domain name. You already have the IP.
>
> > This is effecting the way the message blocks are being generated. Trying
> to
> > call SmbSession.getChallenge( ) fails. The message block in the log looks
> > very corrupted (to me), and the server sends back a rejection message
> > (called name not present).
> >
> > The following command line test class shows this problem well. All it's
> > supposed to do is get a challenge from the domain controller...
> >
> > public static void main(String[] args) throws Exception
> > {
> > jcifs.util.Log.setMask(jcifs.util.Log.ALL);
> > UniAddress domainController = UniAddress.getByName("148.162.36.216",
> > true);
> > byte[] challenge = SmbSession.getChallenge(domainController);
> >
> > System.out.println(domainController);
> > for (int i=0; i < challenge.length; i++)
> > System.out.print(challenge[i] + ":");
> > System.out.println();
> > }
> >
> > Running with the default file.encoding (Cp1047) gives the following output
> > (trimmed for relevance). Notice the messed up host name in the
> exception...
> > ========================= BEGIN OUTPUT
> > ================================================
> > Oct 23 12:06:17.646 - name service address cache
> > JCIFS175_2_??<?É> JCIFS175_2_??<?É>/148.162.175.2
> > 0.0.0.0<ÉÉ> 0.0.0.0<ÉÉ>/0.0.0.0
> >
> > Oct 23 12:06:17.790 - smb negotiation warning
> > requesting negotiation with 0.0.0.0<ÉÉ>/148.162.36.216
> > Oct 23 12:06:17.941 - smb warning
> > ERR_SSN_SRVC/Called name not present
> > Oct 23 12:06:17.964 - nbt name service debugging
> > new data read from socket
> > Oct 23 12:06:17.967 - nbt name service packet sent
> >
> NodeStatusRequest[nameTrnId=1,isResponse=false,opCode=QUERY,isAuthAnswer=fal
> > se,isTruncated=false,isRecurAvailable=false,isRecurDesir
> > Oct 23 12:06:17.969 - datagram packet sent to: 148.162.36.216
> > 00000: 00 01 00 00 00 01 00 00 00 00 00 00 20 43 4B 41 |............ CKA|
> > 00010: 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 |AAAAAAAAAAAAAAAA|
> > 00020: 41 41 41 41 41 41 41 41 41 41 41 41 41 00 00 21 |AAAAAAAAAAAAA..!|
> > 00030: 00 01 |.. |
> >
> > Oct 23 12:06:17.972 - nbt name service packet receviced
> >
> NodeStatusResponse[nameTrnId=1,isResponse=true,opCode=QUERY,isAuthAnswer=tru
> > e,isTruncated=false,isRecurAvailable=false,isRecurDesire...
> > Oct 23 12:06:17.983 - datagram packet received from: 148.162.36.216
> > 00000: 00 01 84 00 00 00 00 01 00 00 00 00 20 43 4B 41 |............ CKA|
> > 00010: 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 |AAAAAAAAAAAAAAAA|
> > 00020: 41 41 41 41 41 41 41 41 41 41 41 41 41 00 00 21 |AAAAAAAAAAAAA..!|
> > 00030: 00 01 00 00 00 00 00 53 02 55 53 4B 49 48 53 56 |.......S.USKIHSV|
> > 00040: 50 57 49 4E 53 43 4C 31 20 44 00 55 53 4B 49 48 |PWINSCL1 D.USKIH|
> > 00050: 53 56 50 57 49 4E 53 43 4C 31 00 44 00 00 03 47 |SVPWINSCL1.D...G|
> > 00060: 78 5A 73 00 00 00 00 00 00 00 00 00 00 00 00 00 |xZs.............|
> > 00070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
> > 00080: 00 00 00 00 00 00 00 00 00 00 00 00 49 00 44 00 |............I.D.|
> > 00090: 5C 00 7B 00 32 00 30 00 38 00 44 00 32 00 43 00 |\.{.2.0.8.D.2.C.|
> > 000A0: 36 00 30 00 2D 00 33 00 41 00 45 00 41 00 2D 00 |6.0.-.3.A.E.A.-.|
> > 000B0: 31 00 30 00 36 00 39 00 2D 00 41 00 32 00 44 00 |1.0.6.9.-.A.2.D.|
> > 000C0: 37 00 2D 00 30 00 38 00 30 00 30 00 32 00 42 00 |7.-.0.8.0.0.2.B.|
> > 000D0: 33 00 30 00 33 00 30 00 39 00 44 00 7D 00 5C 00 |3.0.3.0.9.D.}.\.|
> > 000E0: 49 00 6E 00 50 00 72 00 6F 00 63 00 53 00 65 00 |I.n.P.r.o.c.S.e.|
> > 000F0: 72 00 76 00 65 00 72 00 33 00 32 00 00 00 00 00 |r.v.e.r.3.2.....|
> > 00100: 00 00 65 00 61 00 26 00 30 |..e.a.&.0 |
> >
> > Oct 23 12:06:17.991 - smb warning
> > ERR_SSN_SRVC/Called name not present
> > Exception in thread "main" jcifs.smb.SmbException: Failed to establish
> > session with --.-¦-+&--+--<æ<ÉÉ>/148.162.36.216
> > at jcifs.smb.SmbTransport.send(SmbTransport.java:478)
> > at jcifs.smb.SmbTransport.negotiate(SmbTransport.java:666)
> > at jcifs.smb.SmbSession.getChallenge(SmbSession.java:34)
> > at Test.main(Test.java:11)
> > ========================== END OUTPUT
> > ============================================
> >
> >
> >
> > Running the same program with the java option -Dfile.encoding=UTF-8, or
> > running it on windows machine gives the following correct output...
> >
> > ========================= BEGIN OUTPUT
> > ================================================
> > Oct 23 12:10:27.592 - name service address cache
> > JCIFS175_2_BC<20> JCIFS175_2_BC<20>/148.162.175.2
> > 0.0.0.0<00> 0.0.0.0<00>/0.0.0.0
> >
> > Oct 23 12:10:27.769 - smb negotiation warning
> > requesting negotiation with 0.0.0.0<00>/148.162.36.216
> > Oct 23 12:10:27.962 - session service warning
> > session established ok with 0.0.0.0<00>/148.162.36.216
> > Oct 23 12:10:27.993 - smb sent
> >
> SmbComNegotiate[command=SMB_COM_NEGOTIATE,received=false,errorCode=0x0000000
> > 0,flags=0x0018,flags2=0x8001,tid=0,pid=35199,uid=0,mid=1...
> >
> > Oct 23 12:10:27.996 - smb sent
> > 00000: FF 53 4D 42 72 00 00 00 00 18 01 80 00 00 00 00 |MBr...........|
> > 00010: 00 00 00 00 00 00 00 00 00 00 7F 89 00 00 01 00 |................|
> > 00020: 00 0C 00 02 4E 54 20 4C 4D 20 30 2E 31 32 00 |....NT LM 0.12. |
> >
> > Oct 23 12:10:28.007 - smb transport warning
> > new data read from socket
> > Oct 23 12:10:28.104 - smb packet format warning
> > byteCount=54 but readBytesWireFormat returned 26
> > Oct 23 12:10:28.146 - smb received
> >
> SmbComNegotiateResponse[command=SMB_COM_NEGOTIATE,received=true,errorCode=0x
> > 00000000,flags=0x0098,flags2=0x8001,tid=0,pid=35199,uid=...
> >
> > Oct 23 12:10:28.162 - smb received
> > 00000: FF 53 4D 42 72 00 00 00 00 98 01 80 00 00 00 00 |MBr...........|
> > 00010: 00 00 00 00 00 00 00 00 00 00 7F 89 00 00 01 00 |................|
> > 00020: 11 00 00 03 32 00 01 00 04 41 00 00 00 00 01 00 |....2....A......|
> > 00030: 00 00 00 00 FD F3 00 00 8A 66 E0 1D AF 7A C2 01 |....²=...fa.»z-.|
> > 00040: F0 00 08 36 00 75 B5 CB C0 C7 E1 6B 4E 4B 00 4D |=..6.u¦-+¦ßkNK.M|
> > 00050: 00 53 00 45 00 52 00 56 00 45 00 52 00 53 00 00 |.S.E.R.V.E.R.S..|
> > 00060: 00 55 00 53 00 4B 00 49 00 48 00 53 00 56 00 50 |.U.S.K.I.H.S.V.P|
> > 00070: 00 50 00 52 00 54 00 31 00 00 00 |.P.R.T.1... |
> >
> > 0.0.0.0<00>/148.162.36.216
> > 117:-75:-53:-64:-57:-31:107:78:
> > ========================== END OUTPUT
> > ============================================
> >
> >
> > This shows that it's very possible to make it work in this environment. I
> > only need to get the jcifs code to quit assuming it's in an ASCII
> > environment.
> >
> > As stated before, changing the file encoding for the entire web server is
> > not a viable option. Does anyone know how to make this work as is, or
> will
> > the internal jcifs code have to be modified to specify UTF-8 encoding when
> > it converts its byte arrays to strings (if that's really what the problem
> > is)?
> >
> > p.s.
> > This also shows a small bug in the hex dump code where some of the
> > characters that tried print weren't printable. That makes it looks like
> the
> > " SMB" header is incorrect, but it's actually right.
> >
> > Thanks to all in advance
> > Ted Bigham
> >
>
>
More information about the jcifs
mailing list