xaizek / vifm (License: GPLv2+) (since 2018-12-07)
Vifm is a file manager with curses interface, which provides Vi[m]-like environment for managing objects within file systems, extended with some useful ideas from mutt.
Clone URLs: https://code.reversed.top/user/xaizek/vifm ssh://rocketgit@code.reversed.top/user/xaizek/vifm
v0.13 v0.13-beta v0.14 v0.14-beta v0.14.1 v0.14.2 v0.14.3 v0.14.4 v0.14.5 master
Commit fb57a0a193e8dd03d6e61877a0b86f0d05e0ff09

Update utf8proc from v2.11.3 to v2.12.0
Changes:
* Unicode 18 support. (This includes the modified grapheme-break rule
GB9c in UAX 29, which removes some grapheme breaks for Indic
characters.)
* `options` arguments are changed from an `enum` to `unsigned int`,
since they are generally bitwise "or" of `enum` values; this should
be backwards binary compatible.
* `int *last_boundclass` parameter of `utf8proc_decompose_char` is
changed to a `utf8proc_int32_t *` (affecting only rare systems where
`int` is not 32 bits, on which the `last_boundclass` argument would
have produced incorrect results).
* New `utf8proc_free` function to free memory allocated by utf8proc.
(You can also use your own memory allocator by calling lower-level
APIs.)
* `utf8proc_normalize_utf32` can now handle invalid codepoints >=
0x110000. They are passed through unchanged rather than dropped, and
composition never runs across one.
* Fix `UTF8PROC_CHARBOUND` emitting no 0xff grapheme markers when combined
with `UTF8PROC_COMPOSE` or `UTF8PROC_DECOMPOSE`, a regression in
2.11.3.
Author: xaizek
Author date (UTC): 2026-10-24 11:51
Committer name: xaizek
Committer date (UTC): 2026-10-24 11:51
Parent(s): d42c9c1f92d62b4f264673da47de1a0f019d4351
Signing key: 99DC5E4DB05F6BE2
Tree: bf4c0ff536c5dd87dcfdad24427fc8b095450d61
File Lines added Lines deleted
ChangeLog 1 1
src/utils/utf8proc.c 61 49
src/utils/utf8proc.h 35 21
src/utils/utf8proc_data.inc 9339 9000
File ChangeLog changed (mode: 100644) (index 262f594f4..b975490a8)
76 76 Added "[A]ppend all" conflict resolution option, which appends tails Added "[A]ppend all" conflict resolution option, which appends tails
77 77 of all conflicting regular files without asking again. Patch by rr-. of all conflicting regular files without asking again. Patch by rr-.
78 78
79 Updated utf8proc to v2.11.3.
79 Updated utf8proc to v2.12.0.
80 80
81 81 Made documentation on which :commands can have comments a bit more Made documentation on which :commands can have comments a bit more
82 82 verbose. verbose.
File src/utils/utf8proc.c changed (mode: 100644) (index a515401cb..340a373d2)
29 29 * The original data files are available at * The original data files are available at
30 30 * https://www.unicode.org/Public/UNIDATA/ * https://www.unicode.org/Public/UNIDATA/
31 31 * *
32 * Please notice the copyright statement in the file "utf8proc_data.c".
32 * Please notice the copyright statement in the file "utf8proc_data.inc".
33 33 */ */
34 34
35 35
 
... ... UTF8PROC_DLLEXPORT const char *utf8proc_version(void) {
101 101 } }
102 102
103 103 UTF8PROC_DLLEXPORT const char *utf8proc_unicode_version(void) { UTF8PROC_DLLEXPORT const char *utf8proc_unicode_version(void) {
104 return "17.0.0";
104 return "18.0.0";
105 105 } }
106 106
107 107 UTF8PROC_DLLEXPORT const char *utf8proc_errmsg(utf8proc_ssize_t errcode) { UTF8PROC_DLLEXPORT const char *utf8proc_errmsg(utf8proc_ssize_t errcode) {
 
... ... static utf8proc_bool grapheme_break_extended(int lbc, int tbc, int licb, int tic
294 294 int state_bc, state_icb; /* boundclass and indic_conjunct_break state */ int state_bc, state_icb; /* boundclass and indic_conjunct_break state */
295 295 if (*state == 0) { /* state initialization */ if (*state == 0) { /* state initialization */
296 296 state_bc = lbc; state_bc = lbc;
297 state_icb = licb == UTF8PROC_INDIC_CONJUNCT_BREAK_CONSONANT ? licb : UTF8PROC_INDIC_CONJUNCT_BREAK_NONE;
297 state_icb = licb;
298 298 } }
299 299 else { /* lbc and licb are already encoded in *state */ else { /* lbc and licb are already encoded in *state */
300 300 state_bc = *state & 0xff; // 1st byte of state is bound class state_bc = *state & 0xff; // 1st byte of state is bound class
 
... ... static utf8proc_bool grapheme_break_extended(int lbc, int tbc, int licb, int tic
305 305 !(state_icb == UTF8PROC_INDIC_CONJUNCT_BREAK_LINKER !(state_icb == UTF8PROC_INDIC_CONJUNCT_BREAK_LINKER
306 306 && ticb == UTF8PROC_INDIC_CONJUNCT_BREAK_CONSONANT); // GB9c && ticb == UTF8PROC_INDIC_CONJUNCT_BREAK_CONSONANT); // GB9c
307 307
308 // Special support for GB9c. Don't break between two consonants
309 // separated 1+ linker characters and 0+ extend characters in any order.
310 // After a consonant, we enter LINKER state after at least one linker.
311 if (ticb == UTF8PROC_INDIC_CONJUNCT_BREAK_CONSONANT
312 || state_icb == UTF8PROC_INDIC_CONJUNCT_BREAK_CONSONANT
313 || state_icb == UTF8PROC_INDIC_CONJUNCT_BREAK_EXTEND)
314 state_icb = ticb;
315 else if (state_icb == UTF8PROC_INDIC_CONJUNCT_BREAK_LINKER)
316 state_icb = ticb == UTF8PROC_INDIC_CONJUNCT_BREAK_EXTEND ?
317 UTF8PROC_INDIC_CONJUNCT_BREAK_LINKER : ticb;
308 // Special support for GB9c. Don't break between linker + 0+ extend chars and consonant.
309 // We enter LINKER state after a linker and stay in it for extend chars.
310 state_icb = (state_icb == UTF8PROC_INDIC_CONJUNCT_BREAK_LINKER && ticb == UTF8PROC_INDIC_CONJUNCT_BREAK_EXTEND) ? state_icb : ticb;
318 311
319 312 // Special support for GB 12/13 made possible by GB999. After two RI // Special support for GB 12/13 made possible by GB999. After two RI
320 313 // class codepoints we want to force a break. Do this by resetting the // class codepoints we want to force a break. Do this by resetting the
 
... ... static utf8proc_int32_t seqindex_decode_index(const utf8proc_uint32_t seqindex)
377 370 return seqindex_decode_entry(&entry); return seqindex_decode_entry(&entry);
378 371 } }
379 372
380 static utf8proc_ssize_t seqindex_write_char_decomposed(utf8proc_uint16_t seqindex, utf8proc_int32_t *dst, utf8proc_ssize_t bufsize, utf8proc_option_t options, int *last_boundclass) {
373 static utf8proc_ssize_t seqindex_write_char_decomposed(utf8proc_uint16_t seqindex, utf8proc_int32_t *dst, utf8proc_ssize_t bufsize, unsigned int options, utf8proc_int32_t *last_boundclass) {
381 374 utf8proc_ssize_t written = 0; utf8proc_ssize_t written = 0;
382 375 const utf8proc_uint16_t *entry = &utf8proc_sequences[seqindex & 0x3FFF]; const utf8proc_uint16_t *entry = &utf8proc_sequences[seqindex & 0x3FFF];
383 376 int len = seqindex >> 14; int len = seqindex >> 14;
 
... ... static utf8proc_ssize_t seqindex_write_char_decomposed(utf8proc_uint16_t seqinde
388 381 for (; len >= 0; entry++, len--) { for (; len >= 0; entry++, len--) {
389 382 utf8proc_int32_t entry_cp = seqindex_decode_entry(&entry); utf8proc_int32_t entry_cp = seqindex_decode_entry(&entry);
390 383
391 written += utf8proc_decompose_char(entry_cp, dst ? dst+written : dst,
392 (bufsize > written) ? (bufsize - written) : 0, options,
393 last_boundclass);
394 if (written < 0) return UTF8PROC_ERROR_OVERFLOW;
384 utf8proc_ssize_t decomp_result = utf8proc_decompose_char(entry_cp, dst ? dst+written : dst,
385 (bufsize > written) ? (bufsize - written) : 0, options, last_boundclass);
386 if (decomp_result < 0) return decomp_result;
387 written += decomp_result;
395 388 } }
396 389 return written; return written;
397 390 } }
 
... ... UTF8PROC_DLLEXPORT int utf8proc_charwidth(utf8proc_int32_t c) {
433 426 } }
434 427
435 428 UTF8PROC_DLLEXPORT utf8proc_bool utf8proc_charwidth_ambiguous(utf8proc_int32_t c) { UTF8PROC_DLLEXPORT utf8proc_bool utf8proc_charwidth_ambiguous(utf8proc_int32_t c) {
436 return utf8proc_get_property(c)->ambiguous_width;
429 return (utf8proc_bool) utf8proc_get_property(c)->ambiguous_width;
437 430 } }
438 431
439 432 UTF8PROC_DLLEXPORT utf8proc_category_t utf8proc_category(utf8proc_int32_t c) { UTF8PROC_DLLEXPORT utf8proc_category_t utf8proc_category(utf8proc_int32_t c) {
 
... ... UTF8PROC_DLLEXPORT const char *utf8proc_category_string(utf8proc_int32_t c) {
447 440
448 441 #define utf8proc_decompose_lump(replacement_uc) \ #define utf8proc_decompose_lump(replacement_uc) \
449 442 return utf8proc_decompose_char((replacement_uc), dst, bufsize, \ return utf8proc_decompose_char((replacement_uc), dst, bufsize, \
450 (utf8proc_option_t)(options & ~(unsigned int)UTF8PROC_LUMP), last_boundclass)
443 options & ~(unsigned int)UTF8PROC_LUMP, last_boundclass)
451 444
452 UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_decompose_char(utf8proc_int32_t uc, utf8proc_int32_t *dst, utf8proc_ssize_t bufsize, utf8proc_option_t options, int *last_boundclass) {
445 UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_decompose_char(utf8proc_int32_t uc, utf8proc_int32_t *dst, utf8proc_ssize_t bufsize, unsigned int options, utf8proc_int32_t *last_boundclass) {
453 446 const utf8proc_property_t *property; const utf8proc_property_t *property;
454 447 utf8proc_propval_t category; utf8proc_propval_t category;
455 448 utf8proc_int32_t hangul_sindex; utf8proc_int32_t hangul_sindex;
 
... ... UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_decompose_char(utf8proc_int32_t uc,
539 532
540 533 UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_decompose( UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_decompose(
541 534 const utf8proc_uint8_t *str, utf8proc_ssize_t strlen, const utf8proc_uint8_t *str, utf8proc_ssize_t strlen,
542 utf8proc_int32_t *buffer, utf8proc_ssize_t bufsize, utf8proc_option_t options
535 utf8proc_int32_t *buffer, utf8proc_ssize_t bufsize, unsigned int options
543 536 ) { ) {
544 537 return utf8proc_decompose_custom(str, strlen, buffer, bufsize, options, NULL, NULL); return utf8proc_decompose_custom(str, strlen, buffer, bufsize, options, NULL, NULL);
545 538 } }
546 539
547 540 UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_decompose_custom( UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_decompose_custom(
548 541 const utf8proc_uint8_t *str, utf8proc_ssize_t strlen, const utf8proc_uint8_t *str, utf8proc_ssize_t strlen,
549 utf8proc_int32_t *buffer, utf8proc_ssize_t bufsize, utf8proc_option_t options,
542 utf8proc_int32_t *buffer, utf8proc_ssize_t bufsize, unsigned int options,
550 543 utf8proc_custom_func custom_func, void *custom_data utf8proc_custom_func custom_func, void *custom_data
551 544 ) { ) {
552 545 /* strlen will be ignored, if UTF8PROC_NULLTERM is set in options */ /* strlen will be ignored, if UTF8PROC_NULLTERM is set in options */
 
... ... UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_decompose_custom(
560 553 utf8proc_int32_t uc; utf8proc_int32_t uc;
561 554 utf8proc_ssize_t rpos = 0; utf8proc_ssize_t rpos = 0;
562 555 utf8proc_ssize_t decomp_result; utf8proc_ssize_t decomp_result;
563 int boundclass = UTF8PROC_BOUNDCLASS_START;
556 utf8proc_int32_t boundclass = UTF8PROC_BOUNDCLASS_START;
564 557 while (1) { while (1) {
565 558 if (options & UTF8PROC_NULLTERM) { if (options & UTF8PROC_NULLTERM) {
566 rpos += utf8proc_iterate(str + rpos, -1, &uc);
559 utf8proc_ssize_t nread = utf8proc_iterate(str + rpos, -1, &uc);
567 560 /* checking of return value is not necessary, /* checking of return value is not necessary,
568 561 as 'uc' is < 0 in case of error */ as 'uc' is < 0 in case of error */
569 562 if (uc < 0) return UTF8PROC_ERROR_INVALIDUTF8; if (uc < 0) return UTF8PROC_ERROR_INVALIDUTF8;
570 if (rpos < 0) return UTF8PROC_ERROR_OVERFLOW;
563 if (nread > (utf8proc_ssize_t)SSIZE_MAX - rpos) return UTF8PROC_ERROR_OVERFLOW;
564 rpos += nread;
571 565 if (uc == 0) break; if (uc == 0) break;
572 566 } else { } else {
573 567 if (rpos >= strlen) break; if (rpos >= strlen) break;
 
... ... UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_decompose_custom(
582 576 &boundclass &boundclass
583 577 ); );
584 578 if (decomp_result < 0) return decomp_result; if (decomp_result < 0) return decomp_result;
585 wpos += decomp_result;
586 579 /* prohibiting integer overflows due to too long strings: */ /* prohibiting integer overflows due to too long strings: */
587 if (wpos < 0 ||
588 wpos > (utf8proc_ssize_t)(SSIZE_MAX/sizeof(utf8proc_int32_t)/2))
580 if (decomp_result > (utf8proc_ssize_t)(SSIZE_MAX/sizeof(utf8proc_int32_t)/2) - wpos)
589 581 return UTF8PROC_ERROR_OVERFLOW; return UTF8PROC_ERROR_OVERFLOW;
582 wpos += decomp_result;
590 583 } }
591 584 } }
592 585 if ((options & (UTF8PROC_COMPOSE|UTF8PROC_DECOMPOSE)) && bufsize >= wpos) { if ((options & (UTF8PROC_COMPOSE|UTF8PROC_DECOMPOSE)) && bufsize >= wpos) {
 
... ... UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_decompose_custom(
621 614 return wpos; return wpos;
622 615 } }
623 616
624 UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_normalize_utf32(utf8proc_int32_t *buffer, utf8proc_ssize_t length, utf8proc_option_t options) {
617 UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_normalize_utf32(utf8proc_int32_t *buffer, utf8proc_ssize_t length, unsigned int options) {
625 618 /* UTF8PROC_NULLTERM option will be ignored, 'length' is never ignored */ /* UTF8PROC_NULLTERM option will be ignored, 'length' is never ignored */
626 619 if (options & (UTF8PROC_NLF2LS | UTF8PROC_NLF2PS | UTF8PROC_STRIPCC)) { if (options & (UTF8PROC_NLF2LS | UTF8PROC_NLF2PS | UTF8PROC_STRIPCC)) {
627 620 utf8proc_ssize_t rpos; utf8proc_ssize_t rpos;
 
... ... UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_normalize_utf32(utf8proc_int32_t *b
662 655 utf8proc_ssize_t wpos = 0; utf8proc_ssize_t wpos = 0;
663 656 for (rpos = 0; rpos < length; rpos++) { for (rpos = 0; rpos < length; rpos++) {
664 657 utf8proc_int32_t current_char = buffer[rpos]; utf8proc_int32_t current_char = buffer[rpos];
665 if (current_char < 0) {
666 /* skip grapheme break */
658 if (current_char < 0 || current_char >= 0x110000) {
659 /* grapheme-break sentinel or out-of-range codepoint: pass it through
660 unchanged, but never compose across it, since we cannot inspect it
661 (unsafe_get_property would OOB on utf8proc_stage1table, idx = uc >> 8).
662 utf8proc_reencode drops what cannot be encoded, so this is invisible
663 to utf8proc_map. */
664 buffer[wpos++] = current_char;
665 starter = NULL;
666 starter_property = NULL;
667 max_combining_class = -1;
667 668 continue; continue;
668 669 } }
669 670 const utf8proc_property_t *current_property = unsafe_get_property(current_char); const utf8proc_property_t *current_property = unsafe_get_property(current_char);
 
... ... UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_normalize_utf32(utf8proc_int32_t *b
695 696 } }
696 697 } }
697 698 if (!starter_property) { if (!starter_property) {
699 /* `*starter` is either a Hangul character, or points into
700 the table `utf8proc_combinations_combined`. It is always
701 valid. */
698 702 starter_property = unsafe_get_property(*starter); starter_property = unsafe_get_property(*starter);
699 703 } }
700 704 int idx = starter_property->comb_index; int idx = starter_property->comb_index;
 
... ... UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_normalize_utf32(utf8proc_int32_t *b
742 746 return length; return length;
743 747 } }
744 748
745 UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_reencode(utf8proc_int32_t *buffer, utf8proc_ssize_t length, utf8proc_option_t options) {
749 UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_reencode(utf8proc_int32_t *buffer, utf8proc_ssize_t length, unsigned int options) {
746 750 /* UTF8PROC_NULLTERM option will be ignored, 'length' is never ignored /* UTF8PROC_NULLTERM option will be ignored, 'length' is never ignored
747 751 ASSERT: 'buffer' has one spare byte of free space at the end! */ ASSERT: 'buffer' has one spare byte of free space at the end! */
748 752 length = utf8proc_normalize_utf32(buffer, length, options); length = utf8proc_normalize_utf32(buffer, length, options);
 
... ... UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_reencode(utf8proc_int32_t *buffer,
767 771 } }
768 772
769 773 UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_map( UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_map(
770 const utf8proc_uint8_t *str, utf8proc_ssize_t strlen, utf8proc_uint8_t **dstptr, utf8proc_option_t options
774 const utf8proc_uint8_t *str, utf8proc_ssize_t strlen, utf8proc_uint8_t **dstptr, unsigned int options
771 775 ) { ) {
772 776 return utf8proc_map_custom(str, strlen, dstptr, options, NULL, NULL); return utf8proc_map_custom(str, strlen, dstptr, options, NULL, NULL);
773 777 } }
774 778
775 779 UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_map_custom( UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_map_custom(
776 const utf8proc_uint8_t *str, utf8proc_ssize_t strlen, utf8proc_uint8_t **dstptr, utf8proc_option_t options,
780 const utf8proc_uint8_t *str, utf8proc_ssize_t strlen, utf8proc_uint8_t **dstptr, unsigned int options,
777 781 utf8proc_custom_func custom_func, void *custom_data utf8proc_custom_func custom_func, void *custom_data
778 782 ) { ) {
779 783 utf8proc_int32_t *buffer; utf8proc_int32_t *buffer;
780 784 utf8proc_ssize_t result; utf8proc_ssize_t result;
785 utf8proc_ssize_t bufsize;
781 786 *dstptr = NULL; *dstptr = NULL;
782 result = utf8proc_decompose_custom(str, strlen, NULL, 0, options, custom_func, custom_data);
783 if (result < 0) return result;
784 buffer = (utf8proc_int32_t *) malloc(((utf8proc_size_t)result) * sizeof(utf8proc_int32_t) + 1);
787 bufsize = utf8proc_decompose_custom(str, strlen, NULL, 0, options, custom_func, custom_data);
788 if (bufsize < 0) return bufsize;
789 buffer = (utf8proc_int32_t *) malloc(((utf8proc_size_t)bufsize) * sizeof(utf8proc_int32_t) + 1);
785 790 if (!buffer) return UTF8PROC_ERROR_NOMEM; if (!buffer) return UTF8PROC_ERROR_NOMEM;
786 result = utf8proc_decompose_custom(str, strlen, buffer, result, options, custom_func, custom_data);
791 result = utf8proc_decompose_custom(str, strlen, buffer, bufsize, options, custom_func, custom_data);
787 792 if (result < 0) { if (result < 0) {
788 793 free(buffer); free(buffer);
789 794 return result; return result;
790 795 } }
796 /* a custom_func that returns different results across the two decompose
797 passes can make the second pass longer than the buffer we sized from the
798 first; reencode would then read and write past the allocation, so bail
799 out instead of corrupting the heap */
800 if (result > bufsize) {
801 free(buffer);
802 return UTF8PROC_ERROR_OVERFLOW;
803 }
791 804 result = utf8proc_reencode(buffer, result, options); result = utf8proc_reencode(buffer, result, options);
792 805 if (result < 0) { if (result < 0) {
793 806 free(buffer); free(buffer);
 
... ... UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_map_custom(
804 817
805 818 UTF8PROC_DLLEXPORT utf8proc_uint8_t *utf8proc_NFD(const utf8proc_uint8_t *str) { UTF8PROC_DLLEXPORT utf8proc_uint8_t *utf8proc_NFD(const utf8proc_uint8_t *str) {
806 819 utf8proc_uint8_t *retval; utf8proc_uint8_t *retval;
807 utf8proc_map(str, 0, &retval, (utf8proc_option_t)(UTF8PROC_NULLTERM | UTF8PROC_STABLE |
808 UTF8PROC_DECOMPOSE));
820 utf8proc_map(str, 0, &retval, UTF8PROC_NULLTERM | UTF8PROC_STABLE | UTF8PROC_DECOMPOSE);
809 821 return retval; return retval;
810 822 } }
811 823
812 824 UTF8PROC_DLLEXPORT utf8proc_uint8_t *utf8proc_NFC(const utf8proc_uint8_t *str) { UTF8PROC_DLLEXPORT utf8proc_uint8_t *utf8proc_NFC(const utf8proc_uint8_t *str) {
813 825 utf8proc_uint8_t *retval; utf8proc_uint8_t *retval;
814 utf8proc_map(str, 0, &retval, (utf8proc_option_t)(UTF8PROC_NULLTERM | UTF8PROC_STABLE |
815 UTF8PROC_COMPOSE));
826 utf8proc_map(str, 0, &retval, UTF8PROC_NULLTERM | UTF8PROC_STABLE | UTF8PROC_COMPOSE);
816 827 return retval; return retval;
817 828 } }
818 829
819 830 UTF8PROC_DLLEXPORT utf8proc_uint8_t *utf8proc_NFKD(const utf8proc_uint8_t *str) { UTF8PROC_DLLEXPORT utf8proc_uint8_t *utf8proc_NFKD(const utf8proc_uint8_t *str) {
820 831 utf8proc_uint8_t *retval; utf8proc_uint8_t *retval;
821 utf8proc_map(str, 0, &retval, (utf8proc_option_t)(UTF8PROC_NULLTERM | UTF8PROC_STABLE |
822 UTF8PROC_DECOMPOSE | UTF8PROC_COMPAT));
832 utf8proc_map(str, 0, &retval, UTF8PROC_NULLTERM | UTF8PROC_STABLE | UTF8PROC_DECOMPOSE | UTF8PROC_COMPAT);
823 833 return retval; return retval;
824 834 } }
825 835
826 836 UTF8PROC_DLLEXPORT utf8proc_uint8_t *utf8proc_NFKC(const utf8proc_uint8_t *str) { UTF8PROC_DLLEXPORT utf8proc_uint8_t *utf8proc_NFKC(const utf8proc_uint8_t *str) {
827 837 utf8proc_uint8_t *retval; utf8proc_uint8_t *retval;
828 utf8proc_map(str, 0, &retval, (utf8proc_option_t)(UTF8PROC_NULLTERM | UTF8PROC_STABLE |
829 UTF8PROC_COMPOSE | UTF8PROC_COMPAT));
838 utf8proc_map(str, 0, &retval, UTF8PROC_NULLTERM | UTF8PROC_STABLE | UTF8PROC_COMPOSE | UTF8PROC_COMPAT);
830 839 return retval; return retval;
831 840 } }
832 841
833 842 UTF8PROC_DLLEXPORT utf8proc_uint8_t *utf8proc_NFKC_Casefold(const utf8proc_uint8_t *str) { UTF8PROC_DLLEXPORT utf8proc_uint8_t *utf8proc_NFKC_Casefold(const utf8proc_uint8_t *str) {
834 843 utf8proc_uint8_t *retval; utf8proc_uint8_t *retval;
835 utf8proc_map(str, 0, &retval, (utf8proc_option_t)(UTF8PROC_NULLTERM | UTF8PROC_STABLE |
836 UTF8PROC_COMPOSE | UTF8PROC_COMPAT | UTF8PROC_CASEFOLD | UTF8PROC_IGNORE));
844 utf8proc_map(str, 0, &retval, UTF8PROC_NULLTERM | UTF8PROC_STABLE | UTF8PROC_COMPOSE | UTF8PROC_COMPAT | UTF8PROC_CASEFOLD | UTF8PROC_IGNORE);
837 845 return retval; return retval;
838 846 } }
847
848 UTF8PROC_DLLEXPORT void utf8proc_free(utf8proc_uint8_t *ptr) {
849 free(ptr);
850 }
File src/utils/utf8proc.h changed (mode: 100644) (index 8d9a2e41e..d842f6bcd)
71 71 /** The MAJOR version number (increased when backwards API compatibility is broken). */ /** The MAJOR version number (increased when backwards API compatibility is broken). */
72 72 #define UTF8PROC_VERSION_MAJOR 2 #define UTF8PROC_VERSION_MAJOR 2
73 73 /** The MINOR version number (increased when new functionality is added in a backwards-compatible manner). */ /** The MINOR version number (increased when new functionality is added in a backwards-compatible manner). */
74 #define UTF8PROC_VERSION_MINOR 11
74 #define UTF8PROC_VERSION_MINOR 12
75 75 /** The PATCH version (increased for fixes that do not change the API). */ /** The PATCH version (increased for fixes that do not change the API). */
76 #define UTF8PROC_VERSION_PATCH 3
76 #define UTF8PROC_VERSION_PATCH 0
77 77 /** @} */ /** @} */
78 78
79 79 #include <stdlib.h> #include <stdlib.h>
 
... ... extern "C" {
143 143 #endif #endif
144 144
145 145 /** /**
146 * Option flags used by several functions in the library.
146 * Option flags used by several functions in the library. For
147 * `unsigned int options` arguments, you can pass a bitwise-or of these
148 * enumerated constants.
147 149 */ */
148 150 typedef enum { typedef enum {
149 151 /** The given UTF-8 input is NULL terminated. */ /** The given UTF-8 input is NULL terminated. */
 
... ... typedef enum {
210 212 /** /**
211 213 * Strip unassigned codepoints. * Strip unassigned codepoints.
212 214 */ */
213 UTF8PROC_STRIPNA = (1<<14),
215 UTF8PROC_STRIPNA = (1<<14)
214 216 } utf8proc_option_t; } utf8proc_option_t;
215 217
216 218 /** @name Error codes /** @name Error codes
 
... ... typedef enum {
348 350 UTF8PROC_CATEGORY_CC = 26, /**< Other, control */ UTF8PROC_CATEGORY_CC = 26, /**< Other, control */
349 351 UTF8PROC_CATEGORY_CF = 27, /**< Other, format */ UTF8PROC_CATEGORY_CF = 27, /**< Other, format */
350 352 UTF8PROC_CATEGORY_CS = 28, /**< Other, surrogate */ UTF8PROC_CATEGORY_CS = 28, /**< Other, surrogate */
351 UTF8PROC_CATEGORY_CO = 29, /**< Other, private use */
353 UTF8PROC_CATEGORY_CO = 29 /**< Other, private use */
352 354 } utf8proc_category_t; } utf8proc_category_t;
353 355
354 356 /** Bidirectional character classes. */ /** Bidirectional character classes. */
 
... ... typedef enum {
375 377 UTF8PROC_BIDI_CLASS_LRI = 20, /**< Left-to-Right Isolate */ UTF8PROC_BIDI_CLASS_LRI = 20, /**< Left-to-Right Isolate */
376 378 UTF8PROC_BIDI_CLASS_RLI = 21, /**< Right-to-Left Isolate */ UTF8PROC_BIDI_CLASS_RLI = 21, /**< Right-to-Left Isolate */
377 379 UTF8PROC_BIDI_CLASS_FSI = 22, /**< First Strong Isolate */ UTF8PROC_BIDI_CLASS_FSI = 22, /**< First Strong Isolate */
378 UTF8PROC_BIDI_CLASS_PDI = 23, /**< Pop Directional Isolate */
380 UTF8PROC_BIDI_CLASS_PDI = 23 /**< Pop Directional Isolate */
379 381 } utf8proc_bidi_class_t; } utf8proc_bidi_class_t;
380 382
381 383 /** Decomposition type. */ /** Decomposition type. */
 
... ... typedef enum {
395 397 UTF8PROC_DECOMP_TYPE_SMALL = 13, /**< Small */ UTF8PROC_DECOMP_TYPE_SMALL = 13, /**< Small */
396 398 UTF8PROC_DECOMP_TYPE_SQUARE = 14, /**< Square */ UTF8PROC_DECOMP_TYPE_SQUARE = 14, /**< Square */
397 399 UTF8PROC_DECOMP_TYPE_FRACTION = 15, /**< Fraction */ UTF8PROC_DECOMP_TYPE_FRACTION = 15, /**< Fraction */
398 UTF8PROC_DECOMP_TYPE_COMPAT = 16, /**< Compat */
400 UTF8PROC_DECOMP_TYPE_COMPAT = 16 /**< Compat */
399 401 } utf8proc_decomp_type_t; } utf8proc_decomp_type_t;
400 402
401 403 /** Boundclass property. (TR29) */ /** Boundclass property. (TR29) */
 
... ... typedef enum {
426 428 /* the Extended_Pictographic property is used in the Unicode 11 /* the Extended_Pictographic property is used in the Unicode 11
427 429 grapheme-boundary rules, so we store it in the boundclass field */ grapheme-boundary rules, so we store it in the boundclass field */
428 430 UTF8PROC_BOUNDCLASS_EXTENDED_PICTOGRAPHIC = 19, UTF8PROC_BOUNDCLASS_EXTENDED_PICTOGRAPHIC = 19,
429 UTF8PROC_BOUNDCLASS_E_ZWG = 20, /* UTF8PROC_BOUNDCLASS_EXTENDED_PICTOGRAPHIC + ZWJ */
431 UTF8PROC_BOUNDCLASS_E_ZWG = 20 /* UTF8PROC_BOUNDCLASS_EXTENDED_PICTOGRAPHIC + ZWJ */
430 432 } utf8proc_boundclass_t; } utf8proc_boundclass_t;
431 433
432 434 /** Indic_Conjunct_Break property. (TR44) */ /** Indic_Conjunct_Break property. (TR44) */
 
... ... typedef enum {
434 436 UTF8PROC_INDIC_CONJUNCT_BREAK_NONE = 0, UTF8PROC_INDIC_CONJUNCT_BREAK_NONE = 0,
435 437 UTF8PROC_INDIC_CONJUNCT_BREAK_LINKER = 1, UTF8PROC_INDIC_CONJUNCT_BREAK_LINKER = 1,
436 438 UTF8PROC_INDIC_CONJUNCT_BREAK_CONSONANT = 2, UTF8PROC_INDIC_CONJUNCT_BREAK_CONSONANT = 2,
437 UTF8PROC_INDIC_CONJUNCT_BREAK_EXTEND = 3,
439 UTF8PROC_INDIC_CONJUNCT_BREAK_EXTEND = 3
438 440 } utf8proc_indic_conjunct_break_t; } utf8proc_indic_conjunct_break_t;
439 441
440 442 /** /**
 
... ... UTF8PROC_DLLEXPORT const utf8proc_property_t *utf8proc_get_property(utf8proc_int
530 532 * - @ref UTF8PROC_STRIPNA - remove unassigned codepoints * - @ref UTF8PROC_STRIPNA - remove unassigned codepoints
531 533 * @param last_boundclass * @param last_boundclass
532 534 * Pointer to an integer variable containing * Pointer to an integer variable containing
533 * the previous codepoint's (boundclass + indic_conjunct_break << 1) if the @ref UTF8PROC_CHARBOUND
535 * the previous codepoint's (boundclass + (indic_conjunct_break << 8)) if the @ref UTF8PROC_CHARBOUND
534 536 * option is used. If the string is being processed in order, this can be initialized to 0 for * option is used. If the string is being processed in order, this can be initialized to 0 for
535 537 * the beginning of the string, and is thereafter updated automatically. Otherwise, this parameter is ignored. * the beginning of the string, and is thereafter updated automatically. Otherwise, this parameter is ignored.
536 538 * *
 
... ... UTF8PROC_DLLEXPORT const utf8proc_property_t *utf8proc_get_property(utf8proc_int
549 551 */ */
550 552 UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_decompose_char( UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_decompose_char(
551 553 utf8proc_int32_t codepoint, utf8proc_int32_t *dst, utf8proc_ssize_t bufsize, utf8proc_int32_t codepoint, utf8proc_int32_t *dst, utf8proc_ssize_t bufsize,
552 utf8proc_option_t options, int *last_boundclass
554 unsigned int options, utf8proc_int32_t *last_boundclass
553 555 ); );
554 556
555 557 /** /**
 
... ... UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_decompose_char(
571 573 */ */
572 574 UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_decompose( UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_decompose(
573 575 const utf8proc_uint8_t *str, utf8proc_ssize_t strlen, const utf8proc_uint8_t *str, utf8proc_ssize_t strlen,
574 utf8proc_int32_t *buffer, utf8proc_ssize_t bufsize, utf8proc_option_t options
576 utf8proc_int32_t *buffer, utf8proc_ssize_t bufsize, unsigned int options
575 577 ); );
576 578
577 579 /** /**
 
... ... UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_decompose(
582 584 */ */
583 585 UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_decompose_custom( UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_decompose_custom(
584 586 const utf8proc_uint8_t *str, utf8proc_ssize_t strlen, const utf8proc_uint8_t *str, utf8proc_ssize_t strlen,
585 utf8proc_int32_t *buffer, utf8proc_ssize_t bufsize, utf8proc_option_t options,
587 utf8proc_int32_t *buffer, utf8proc_ssize_t bufsize, unsigned int options,
586 588 utf8proc_custom_func custom_func, void *custom_data utf8proc_custom_func custom_func, void *custom_data
587 589 ); );
588 590
 
... ... UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_decompose_custom(
605 607 * @return * @return
606 608 * In case of success, the length (in codepoints) of the normalized UTF-32 string is * In case of success, the length (in codepoints) of the normalized UTF-32 string is
607 609 * returned; otherwise, a negative error code is returned (utf8proc_errmsg()). * returned; otherwise, a negative error code is returned (utf8proc_errmsg()).
608 *
609 * @warning The entries of the array pointed to by `str` have to be in the
610 * range `0x0000` to `0x10FFFF`. Otherwise, the program might crash!
611 610 */ */
612 UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_normalize_utf32(utf8proc_int32_t *buffer, utf8proc_ssize_t length, utf8proc_option_t options);
611 UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_normalize_utf32(utf8proc_int32_t *buffer, utf8proc_ssize_t length, unsigned int options);
613 612
614 613 /** /**
615 614 * Reencodes the sequence of `length` codepoints pointed to by `buffer` * Reencodes the sequence of `length` codepoints pointed to by `buffer`
 
... ... UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_normalize_utf32(utf8proc_int32_t *b
639 638 * entries of the array pointed to by `str` have to be in the * entries of the array pointed to by `str` have to be in the
640 639 * range `0x0000` to `0x10FFFF`. Otherwise, the program might crash! * range `0x0000` to `0x10FFFF`. Otherwise, the program might crash!
641 640 */ */
642 UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_reencode(utf8proc_int32_t *buffer, utf8proc_ssize_t length, utf8proc_option_t options);
641 UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_reencode(utf8proc_int32_t *buffer, utf8proc_ssize_t length, unsigned int options);
643 642
644 643 /** /**
645 644 * Given a pair of consecutive codepoints, return whether a grapheme break is * Given a pair of consecutive codepoints, return whether a grapheme break is
 
... ... UTF8PROC_DLLEXPORT const char *utf8proc_category_string(utf8proc_int32_t codepoi
750 749 * *
751 750 * @note The memory of the new UTF-8 string will have been allocated * @note The memory of the new UTF-8 string will have been allocated
752 751 * with `malloc`, and should therefore be deallocated with `free`. * with `malloc`, and should therefore be deallocated with `free`.
752 * However, it is safer to deallocate it with @ref utf8proc_free in
753 * case your application is linked to a different C library than utf8proc.
753 754 * *
754 755 * @note `utf8proc_map` simply calls `utf8proc_decompose` followed by `utf8proc_reencode`, * @note `utf8proc_map` simply calls `utf8proc_decompose` followed by `utf8proc_reencode`,
755 756 * and applications requiring greater control over memory allocation should instead call * and applications requiring greater control over memory allocation should instead call
756 757 * those two functions directly. * those two functions directly.
757 758 */ */
758 759 UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_map( UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_map(
759 const utf8proc_uint8_t *str, utf8proc_ssize_t strlen, utf8proc_uint8_t **dstptr, utf8proc_option_t options
760 const utf8proc_uint8_t *str, utf8proc_ssize_t strlen, utf8proc_uint8_t **dstptr, unsigned int options
760 761 ); );
761 762
762 763 /** /**
763 * Like utf8proc_map(), but also takes a `custom_func` mapping function
764 * Like @ref utf8proc_map, but also takes a `custom_func` mapping function
764 765 * that is called on each codepoint in `str` before any other transformations * that is called on each codepoint in `str` before any other transformations
765 766 * (along with a `custom_data` pointer that is passed through to `custom_func`). * (along with a `custom_data` pointer that is passed through to `custom_func`).
766 767 * The `custom_func` argument is ignored if it is `NULL`. * The `custom_func` argument is ignored if it is `NULL`.
767 768 */ */
768 769 UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_map_custom( UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_map_custom(
769 const utf8proc_uint8_t *str, utf8proc_ssize_t strlen, utf8proc_uint8_t **dstptr, utf8proc_option_t options,
770 const utf8proc_uint8_t *str, utf8proc_ssize_t strlen, utf8proc_uint8_t **dstptr, unsigned int options,
770 771 utf8proc_custom_func custom_func, void *custom_data utf8proc_custom_func custom_func, void *custom_data
771 772 ); );
772 773
 
... ... UTF8PROC_DLLEXPORT utf8proc_ssize_t utf8proc_map_custom(
776 777 * NFKC_Casefold normalized version of the null-terminated string `str`. These * NFKC_Casefold normalized version of the null-terminated string `str`. These
777 778 * are shortcuts to calling utf8proc_map() with @ref UTF8PROC_NULLTERM * are shortcuts to calling utf8proc_map() with @ref UTF8PROC_NULLTERM
778 779 * combined with @ref UTF8PROC_STABLE and flags indicating the normalization. * combined with @ref UTF8PROC_STABLE and flags indicating the normalization.
780 *
781 * @note The memory of the new UTF-8 string will have been allocated
782 * with `malloc`, and should therefore be deallocated with `free`.
783 * However, it is safer to deallocate it with @ref utf8proc_free in
784 * case your application is linked to a different C library than utf8proc.
779 785 */ */
780 786 /** @{ */ /** @{ */
781 787 /** NFD normalization (@ref UTF8PROC_DECOMPOSE). */ /** NFD normalization (@ref UTF8PROC_DECOMPOSE). */
 
... ... UTF8PROC_DLLEXPORT utf8proc_uint8_t *utf8proc_NFKC(const utf8proc_uint8_t *str);
793 799 UTF8PROC_DLLEXPORT utf8proc_uint8_t *utf8proc_NFKC_Casefold(const utf8proc_uint8_t *str); UTF8PROC_DLLEXPORT utf8proc_uint8_t *utf8proc_NFKC_Casefold(const utf8proc_uint8_t *str);
794 800 /** @} */ /** @} */
795 801
802 /**
803 * Deallocate memory allocated and returned by @ref utf8proc_map and similar functions
804 * (which simply calls the `free` function from the underlying C library linked to utf8proc).
805 * It is safer to call `utf8proc_free` than calling `free` directly, in case your application
806 * is linked to a different C library with incompatible `malloc` and `free` functions.
807 */
808 UTF8PROC_DLLEXPORT void utf8proc_free(utf8proc_uint8_t *ptr);
809
796 810 #ifdef __cplusplus #ifdef __cplusplus
797 811 } }
798 812 #endif #endif
The diff for file src/utils/utf8proc_data.inc is too big (18339 changes) and cannot be shown.
Hints
Before first commit, do not forget to setup your git environment:
git config --global user.name "your_name_here"
git config --global user.email "your@email_here"

Clone this repository using HTTP(S):
git clone https://code.reversed.top/user/xaizek/vifm

Clone this repository using ssh (do not forget to upload a key first):
git clone ssh://rocketgit@code.reversed.top/user/xaizek/vifm

You are allowed to anonymously push to this repository.
This means that your pushed commits will automatically be transformed into a pull request:
... clone the repository ...
... make some changes and some commits ...
git push origin master