TOPIC:

X# STRCONV() implementation proposal 04 Feb 2021 20:36 #17403

atlopes
Topic Author

Offline

Posts: 84

The STRCONV function is a notorious miss from the VFP.Net toolkit list of implemented functions.

This is a proposal for its implementation. It tries to accommodate the difference in the nature of VFP and X# (.Net) strings. VFP strings are made of 8 bit ANSI or binary characters, X# strings are Unicode.

The main challenge is to provide the function's nuclear functionality while adding some value to the X# environment.

Some key points

Conversion to Unicode or to DBCS strings return strings.
Base64 and hexadecimal representations of data are treated as strings.
Single-byte strings and UTF-8 are returned as Binary.
Regional identifiers (may) affect single-byte and DBCS strings.
Added two types that represent a Unicode string as a Binary (for LE and BE).

Probably something else will come up with the discussion to follow.

USING System.Text   
USING System.IO
USING System.Globalization
USING XSharp.Core

FUNCTION Start() AS VOID 

    LOCAL TextNr AS Int
    
    TestNr = 1

    // Base64
    QuickTest(TestNr++, 'StrConv("Abcd", STRCNV_SB_BASE64)', StrConv("Abcd", STRCNV_SB_BASE64), "QWJjZA==")
    QuickTest(TestNr++, 'StrConv("QWJjZA==", STRCNV_BASE64_SB)', StrConv("QWJjZA==", STRCNV_BASE64_SB), (Binary)"Abcd")
    QuickTest(TestNr++, 'StrConv(0h010203fdfeff, STRCNV_SB_BASE64)', StrConv(0h010203fdfeff, STRCNV_SB_BASE64), "AQID/f7/")
    QuickTest(TestNr++, 'StrConv("AQID/f7/", STRCNV_BASE64_SB)', StrConv("AQID/f7/", STRCNV_BASE64_SB), 0h010203fdfeff)

    // Hex
    QuickTest(TestNr++, 'StrConv("Abcd", STRCNV_SB_HEX)', StrConv("Abcd", STRCNV_SB_HEX), "41626364")
    QuickTest(TestNr++, 'StrConv("41626364", STRCNV_HEX_SB)', StrConv("41626364", STRCNV_HEX_SB), (Binary)"Abcd")
    QuickTest(TestNr++, 'StrConv(0h010203fdfeff, STRCNV_SB_HEX)', StrConv(0h010203fdfeff, STRCNV_SB_HEX), "010203FDFEFF")
    QuickTest(TestNr++, 'StrConv("010203FDFEFF", STRCNV_HEX_SB)', StrConv("010203fdfeff", STRCNV_HEX_SB), 0h010203fdfeff)

    // Unicode
    QuickTest(TestNr++, 'StrConv(StrConv(StrConv("Привет мир!", STRCNV_UNI_DBCS, 1251, 1), STRCNV_DBCS_SB), STRCNV_SB_HEX)', ;
        StrConv(StrConv(StrConv("Привет мир!", STRCNV_UNI_DBCS, 1251, 1), STRCNV_DBCS_SB), STRCNV_SB_HEX), "CFF0E8E2E5F220ECE8F021")
    QuickTest(TestNr++, 'StrConv("Привет мир!", STRCNV_UNI_UTF8)', StrConv("Привет мир!", STRCNV_UNI_UTF8), 0hD09FD180D0B8D0B2D0B5D18220D0BCD0B8D18021)
    QuickTest(TestNr++, 'StrConv(e"Bel\xc3\xa9m do Par\xc3\xa1", STRCNV_UTF8_UNI)', StrConv(e"Bel\xc3\xa9m do Par\xc3\xa1", STRCNV_UTF8_UNI), "Belém do Pará")
    QuickTest(TestNr++, 'StrConv(0hC8, STRCNV_DBCS_UNI, 1253, 1) + StrConv(0hC8, STRCNV_DBCS_UNI, 1256, 1)', ;
        StrConv(0hC8, STRCNV_DBCS_UNI, 1253, 1) + StrConv(0hC8, STRCNV_DBCS_UNI, 1256, 1), 'Θب')
    // Extras
    QuickTest(TestNr++, 'StrConv("Γειά σου Κόσμε!", STRCNV_UNI_SB)', StrConv("Γειά σου Κόσμε!", STRCNV_UNI_SB), 0h9303B503B903AC032000C303BF03C50320009A03CC03C303BC03B5032100)

    WAIT
    
	RETURN 

FUNCTION QuickTest (Test AS Int, Expression AS String, Result AS USUAL, Expected AS USUAL) AS Void
    
    ? Test, Expression, "->", Result, "(" + IIF(Result == Expected, "Success", "Fail, was expecting " + Expected) + ")"

ENDFUNC

DEFINE STRCNV_SB_DBCS           = 1
DEFINE STRCNV_DBCS_SB           = 2
DEFINE STRCNV_KATA_HIRA         = 3
DEFINE STRCNV_HIRA_KATA         = 4
DEFINE STRCNV_DBCS_UNI          = 5
DEFINE STRCNV_UNI_DBCS          = 6
DEFINE STRCNV_LOWER             = 7
DEFINE STRCNV_UPPER             = 8
DEFINE STRCNV_DBCS_UTF8         = 9
DEFINE STRCNV_UNI_UTF8          = 10
DEFINE STRCNV_UTF8_DBCS         = 11
DEFINE STRCNV_UTF8_UNI          = 12
DEFINE STRCNV_SB_BASE64         = 13
DEFINE STRCNV_BASE64_SB         = 14
DEFINE STRCNV_SB_HEX            = 15
DEFINE STRCNV_HEX_SB            = 16
DEFINE STRCNV_UNI_SB            = 17
DEFINE STRCNV_UNIBE_SB          = 18

DEFINE STRCNV_ID_LCID           = 0
DEFINE STRCNV_ID_CODEPAGE       = 1
DEFINE STRCNV_ID_CHARSET        = 2

FUNCTION StrConv (Expression AS String, ConversionSetting AS Int, RegionalIdentifier = 0 AS Int, RegionalIDType = STRCNV_ID_LCID AS Int) AS USUAL

    RETURN StrConv_Helper(Expression, (Binary)Expression, ConversionSetting, RegionalIdentifier, RegionalIDType)

END FUNCTION

FUNCTION StrConv (Expression AS Binary, ConversionSetting AS Int, RegionalIdentifier = 0 AS Int, RegionalIDType = STRCNV_ID_LCID AS Int) AS USUAL

    RETURN StrConv_Helper((String)Expression, Expression, ConversionSetting, RegionalIdentifier, RegionalIDType)

END FUNCTION

FUNCTION StrConv (Expression AS USUAL, ConversionSetting AS Int, RegionalIdentifier = 0 AS Int, RegionalIDType = STRCNV_ID_LCID AS Int) AS USUAL

    SWITCH VARTYPE(Expression)
        CASE "C"
            LOCAL StrExpression AS String
        
            StrExpression = Expression
            RETURN StrConv(StrExpression, ConversionSetting, RegionalIdentifier, RegionalIDType)

        CASE "Q"
            LOCAL BinExpression AS Binary
        
            BinExpression = Expression
            RETURN StrConv(BinExpression, ConversionSetting, RegionalIdentifier, RegionalIDType)

        OTHERWISE
            THROW ArgumentException{"Incorrect parameter.", nameof(Expression)}
    END SWITCH

END FUNCTION
 
FUNCTION StrConvStr (Expression AS Binary, RegionalIdentifier = 0 AS Int, RegionalIDType = STRCNV_ID_LCID AS Int) AS String

    VAR enc = StrConv_GetEncoding(RegionalIdentifier, RegionalIDType)
    RETURN Encoding.Unicode.GetString(Encoding.Convert(enc, Encoding.Unicode, Expression))
            
END FUNCTION

STATIC FUNCTION StrConv_helper (Expression AS String, BinaryExpression AS Binary, ConversionSetting AS Int, RegionalIdentifier = 0 AS Int, RegionalIDType = STRCNV_ID_LCID AS Int) AS USUAL
    
    LOCAL ReturnStr AS String
    LOCAL ReturnBin AS Binary
    LOCAL ReturnType AS String

    LOCAL defenc = Encoding.Default AS System.Text.Encoding
    LOCAL utf8 = Encoding.UTF8 AS System.Text.Encoding
    LOCAL unicode = Encoding.Unicode AS System.Text.Encoding
    LOCAL enc AS System.Text.Encoding

    LOCAL Indexer AS Int

    SWITCH (ConversionSetting)

        CASE STRCNV_SB_DBCS
             IF RegionalIDType == STRCNV_ID_LCID
                enc = StrConv_GetEncoding(RegionalIdentifier, STRCNV_ID_LCID)
                ReturnStr = defenc.GetString(Encoding.Convert(defenc, enc, BinaryExpression))
                ReturnType = "S"
            ELSE
                THROW ArgumentException {}
            END IF

        CASE STRCNV_DBCS_SB
             IF RegionalIDType == STRCNV_ID_LCID
                enc = StrConv_GetEncoding(RegionalIdentifier, STRCNV_ID_LCID)
                ReturnBin = Encoding.Convert(defenc, enc, BinaryExpression)
                ReturnType = "B"
            ELSE
                THROW ArgumentException {}
            END IF
                
        CASE STRCNV_KATA_HIRA
            ReturnStr = Expression       // not implemented
            ReturnType = "S"

        CASE STRCNV_HIRA_KATA
            ReturnStr = Expression       // not implemented
            ReturnType = "S"

        CASE STRCNV_DBCS_UNI
            enc = StrConv_GetEncoding(RegionalIdentifier, RegionalIDType)
            ReturnStr = unicode.GetString(Encoding.Convert(enc, unicode, BinaryExpression))
            ReturnType = "S"

        CASE STRCNV_UNI_DBCS
            enc = StrConv_GetEncoding(RegionalIdentifier, RegionalIDType)
            ReturnStr = defenc.GetString(Encoding.Convert(unicode, enc, unicode.GetBytes(Expression)))
            ReturnType = "S"

        CASE STRCNV_LOWER
            IF RegionalIDType == STRCNV_ID_LCID
                ReturnBin = defenc.GetBytes(defenc.GetString(BinaryExpression).ToLower(StrConv_GetCulture(RegionalIdentifier)))
                ReturnType = "B"
            ELSE
                THROW ArgumentException {}
            END IF

        CASE STRCNV_UPPER
             IF RegionalIDType == STRCNV_ID_LCID
                ReturnBin = defenc.GetBytes(defenc.GetString(BinaryExpression).ToUpper(StrConv_GetCulture(RegionalIdentifier)))
                ReturnType = "B"
            ELSE
                THROW ArgumentException {}
            END IF

        CASE STRCNV_DBCS_UTF8
            enc = StrConv_GetEncoding(RegionalIdentifier, RegionalIDType)
            ReturnBin = Encoding.Convert(enc, Encoding.UTF8, BinaryExpression)
            ReturnType = "B"

        CASE STRCNV_UNI_UTF8
            ReturnBin = Encoding.Convert(unicode, utf8, unicode.GetBytes(Expression))
            ReturnType = "B"
            
        CASE STRCNV_UTF8_DBCS
            enc = StrConv_GetEncoding(RegionalIdentifier, RegionalIDType)
            ReturnStr = defenc.GetString(Encoding.Convert(Encoding.UTF8, enc, BinaryExpression))
            ReturnType = "S"

        CASE STRCNV_UTF8_UNI
            ReturnStr = unicode.GetString(Encoding.Convert(Encoding.UTF8, Encoding.Unicode, BinaryExpression))
            ReturnType = "S"
            
        CASE STRCNV_SB_BASE64
            ReturnStr = Convert.ToBase64String(BinaryExpression)
            ReturnType = "S"
            
        CASE STRCNV_BASE64_SB
            ReturnBin = Convert.FromBase64String(defenc.GetString(BinaryExpression))
            ReturnType = "B"

        CASE STRCNV_SB_HEX
            ReturnStr = SUBSTR(BinaryExpression.ToString(), 3)
            ReturnType = "S"

        CASE STRCNV_HEX_SB
            VAR SingleBytes = Byte[]{Expression.Length / 2}
            LOCAL Hex AS Int
            Hex = 0
            FOR Indexer = 1 TO SingleBytes.Length
                SingleBytes[Indexer] = Convert.ToByte(Expression.Substring(Hex, 2), 16)
                Hex += 2
            NEXT
            ReturnBin = SingleBytes
            ReturnType = "B"
            
        CASE STRCNV_UNI_SB
            ReturnBin = unicode.GetBytes(Expression)
            ReturnType = "B"

        CASE STRCNV_UNIBE_SB
            ReturnBin = Encoding.BigEndianUnicode.GetBytes(Expression)
            ReturnType = "B"

        OTHERWISE
            THROW ArgumentException {}
            
    END SWITCH

    SWITCH ReturnType
        CASE "S"
            RETURN ReturnStr
        CASE "B"
            RETURN ReturnBin
        END SWITCH

END FUNCTION

STATIC FUNCTION StrConv_GetEncoding (RegionalIdentifier AS Int, RegionalIDType AS Int) AS Encoding
    
    SWITCH RegionalIDType
        
        CASE STRCNV_ID_LCID
            VAR culture = StrConv_GetCulture(RegionalIdentifier)
            LOCAL CP = culture.TextInfo.ANSICodePage AS Int
 
            RETURN Encoding.GetEncoding(CP)

        CASE STRCNV_ID_CODEPAGE
            RETURN Encoding.GetEncoding(RegionalIdentifier)

        CASE STRCNV_ID_CHARSET
            LOCAL CP = 1252 AS Int
            SWITCH RegionalIdentifier
                CASE 128
                    CP = 932
                CASE 129
                    CP = 949
                CASE 130
                    CP = 1361
                CASE 134
                    CP = 936
                CASE 136
                    CP = 950
                CASE 161
                    CP = 1253
                CASE 162
                    CP = 1254
                CASE 163
                    CP = 1258
                CASE 177
                    CP = 1255
                CASE 178
                    CP = 1256
                CASE 186
                    CP = 1257
                CASE 204
                    CP = 1251
                CASE 222
                    CP = 874
                CASE 238
                    CP = 1250
            END SWITCH
            RETURN Encoding.GetEncoding(CP)
    END SWITCH

    RETURN Encoding.Default

END FUNCTION

STATIC FUNCTION StrConv_GetCulture (LCID AS Int) AS CultureInfo
    
    IF LCID != 0
        RETURN CultureInfo.GetCultureInfo(LCID)
    ELSE
        RETURN CultureInfo.CurrentCulture
    END IF
    
END FUNCTION

Please Log in or Create an account to join the conversation.

X# STRCONV() implementation proposal 04 Feb 2021 21:27 #17404

robert

Offline

Posts: 3446

Antonio,
Thanks for the proposal.
I will refrain from comments for now: I have no idea what FoxPro does with this function so I can't judge the implementation.
But what you have done looks very well thought through.
Maybe some small suggestions:
- the default codepage in StrConv_GetEncoding could be XSharp.RuntimeState.WinCodepage (this is read from the OS).
- the argument exceptions could probably include a message
And we will most certainly copy the "QuickTest" code and add it to our unit tests.

Robert

XSharp Development Team
The Netherlands

Please Log in or Create an account to join the conversation.

Last edit: by robert.

X# STRCONV() implementation proposal 05 Feb 2021 09:58 #17408

atlopes
Topic Author

Offline

Posts: 84

Robert,

I hope VFP developers will step in, test the implementation, point out its problems, and propose corrections and enhancements.

The "final" version, if we come to that, it's for the X# team to integrate with any adjustments you will find fit. As we discussed, that version will be submitted as a pull request.

Please Log in or Create an account to join the conversation.

X# STRCONV() implementation proposal 05 Feb 2021 19:14 #17417

Eric Selje

Offline

Posts: 31

I think this is very cool. I'm trying to figure out a way to create tests so that rather than compare against an "expected value" it would compare against what VFP returned so we could ensure they're the same. If we compiled this into a .NET Class we could call that from DotNetBridge in VFP and have the tests in FoxUnit. That'd be the real test whether it's compatible.

StrConv() is an odd function in VFP and mostly obsolete because of the native functionality in .NET, but we're going to need it included if we want our apps to compile and run correctly.

Please Log in or Create an account to join the conversation.

X# STRCONV() implementation proposal 05 Feb 2021 20:51 #17419

atlopes
Topic Author

Offline

Posts: 84

Eric,

The expected results in tests come from values returned by VFP, but going through DotNetBridge and FoxUnit will be much more stressful and, thus, clarifying. I'm looking forward to it. And surely it will serve as a foundation for tests of the implementation of other VFP functions.

Please Log in or Create an account to join the conversation.

Page:
1

Forum

Public Forums

Visual FoxPro

X# STRCONV() implementation proposal